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SUMMARY 


Performance  Monitoring,  Fault  Detection  and  Fault  Localization 
(PM/FD/FL)  in  large  complex  systems  is  a  growing  discipline  that  overwhelms 
the  more  familiar  Built  In  Test  (BIT)  process.  At  present,  this  field  is 
generally  underestimated  and  unstandardized.  In  several  recent  instances, 
lack  of  a  detailed  design  approach  has  led  to  unsatisfactory  add-on 
PM/FD/FL  functions  to  systems. 

This  report  contains  a  standardized  approach  to  the  acquisition  of 
PM/FD/FL  functions  and  a  discussion  of  the  optimization  of  these  functions. 
It  is  expected  that  a  standardized  approach  will  ensure  the  development  of 
acceptable  PM/FD/FL  functions  within  budget,  schedule,  and  in  phase  with 
other  system  development  efforts.  The  discussion  will  instill  an 
appreciation  for  the  complexity  of  PM/FD/FL  development  and  provide 
assistance  in  optimizing  the  process. 

An  introduction  to  PM/FD/FL,  an  in-depth  plan  for  development  program 
elements  and  tasks,  and  several  appendices, (A  through  E)  including  one  on 
the  optimization  of  PM/FD/FL  development  is  presented. 
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PERFORMANCE  MONITORING,  FAULT  DETECTION,  FAULT  LOCALIZATION  DESIGN  GUIDANCE 

SECTION  I  INTRODUCTION 

ADVERSE  TRENDS 

The  complexity,  size,  required  performance,  and  the  need  for  rapid 
recovery  from  hardware  and  software  malfunctions  of  our  military  systems 
are  steadily  increasing,  while  both  the  manning  and  skill  levels  of  our 
operating  forces  are  decreasing.  This  combination  of  events  may  result  in 
unmanageable  maintenance  and  logistics  burdens  unless  compensating 
provisions  are  introduced.  One  possible  compensating  provision  is  the 
inclusion  of  highly  effective  Performance  Monitoring,  Fault  Detection,  and 
Fault  Localization  (PM/FD^FL)  functions  in  new  systems.  The  fielding  of 
some  recent  systems  containing  far  from  optimal  PM/FD/FL  functions 
indicates  that  this  will  not  be  a  compensating  provision  unless 
improvements  are  made  in  the  way  PM/FD/FL  functions  are  developed. 

Experience  has  shown  that  after  systems  become  fully  operational  a 
number  of  problems  are  likely  to  emerge,  and  there  must  be  an  orderly, 
timely,  and  cost  effective  manner  of  handling  them.  Problems  continue  to 
emerge  as  systems  mature,  and  many  of  these  pose  almost  insurmountable 
difficulties  to  the  maintenance  technicians.  The  less  capable  the  PM/FD/FL 
functions,  the  more  difficult  these  problems  become.  These  problems  are 
made  even  more  difficult  by  the  rapid  turn-over  of  highly  skilled 
maintenance  personnel. 

REVERSING  THE  TRENDS 

The  threatening  trends  can  be  minimized  by  the  incorporation  of  highly 
effective  automated  PM/FD/FL  functions  in  complex  systems.  The  problem  is 
to  select  a  methodology  that  ensures  effectiveness.  This  report  contains  a 
standardized  methodology  for  ensuring  development  of  highly  effective 
PM/FD/FL  functions.  It  also  contains  some  guidance  on  the  selection  of 
PM/FD/FL  parameters  and  optimization  of  the  functions.  Application  of  the 
standardized  methodology  and  the  optimization  suggestions  should 
significantly  reduce  the  previously  cited  maintenance  and  logistic  burdens. 

PM/FD/FL  DESCRIPTION 

PM/FD/FL  is  a  sophisticated  computer  controlled  three-function  type  of 
Built  In  Test  (BIT) .  It  consists  of  methods  that  ensure  system  integrity 
and  consistency  and  eliminate  the  need  for  the  operator  to  make  value 
judgments.  System  integrity  is  determined  by  the  design  of  the  PM/FD/FL 
functions  -  not  by  operator  motivation,  training,  or  ability.  PM/FD/FL  must 
be  incorporated  into  equipment  and  systems  where  construction  complexity, 
environment,  or  required  response  times  cause  the  unaided  determination  of 
system  integrity  or  the  identification  and  isolation  of  faults  to  be  beyond 
the  capability  of  available  personnel. 
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In  practice,  the  three  PM/FD/FL  functions  are  often  independent  functions, 
each  with  a  level  of  sophistication  commensurate  with  the  needs  of  the 
application.  They  are  also  often  misunderstood.  The  following  explanatory 
definitions  are  intended  to  provide  a  better  understanding  of  each  of  the 
PM/FD/FL  functions. 

PERFORMANCE  MONITORING  (PM)  is  the  function  that  determines 
the  integrity  of  an  equipment  or  system.  It  accomplishes  this  by 
injecting  known  and  quantified  inputs  into  the  equipment  or  system, 
observing  responses  for  expected  performance,  and  reporting 
deviations  from  expected  performance.  The  injection  of  test  signals 
is  usually  planned  so  that  they  do  not  interfere  with  the  operation 
of  the  equipment  or  system.  The  PM  function  may  also  detect  faults, 
but  unless  the  application  allows  the  combination  of  PM  and  FD 
functions,  fault  detection  by  the  PM  function  is  a  secondary 
requirement.  PM  is  necessary  because  deviations  from  the  tolerances  of 
individual  items  within  a  system  could  be  acceptable,  but  can  combine  to 
produce  unacceptable  degradation  of  the  total  system;  and  because  trends 
and/or  tendencies  to  eventual  failure  would  most  likely  be  detected  through 
PM. 


FAULT  DETECTION  (FD)  is  the  function  that  detects  and  reports 
faults.  It  is  usually  based  on  the  monitoring  of  normally  occurring 
outputs  at  selected  test  points  for  expected  observations.  During 
normal  operation  of  the  equipment  or  system,  there  is  no  test  signal 
injection.  FD  may,  in  some  cases,  be  combined  with  PM,  but  FD 
cannot  be  used  to  perform  PM.  The  absence  of  faults  is  not  prima 
facie  evidence  that  the  system  is  working  correctly.  FD  requirements  are 
to  detect  all  measurable  faults,  including  those  that  do 
not  cause  immediate  equipment  or  system  failure  but  may  ultimately 
result  in  system  failure.  These  are  defined  as  eminent  failures. 

The  failure  definitions  in  the  equipment/ system  specification  should 
define  the  faults  that  cause  immediate  system  failure  and  those 
that  could  cause  eminent  failures. 

FAULT  LOCALIZATION  (FL)  is  the  function  that  isolates  faults 
found  by  the  PM  and  FD  functions  down  to  a  group  of  modules  that  is 
small  enough  for  effective  maintenance  at  the  organizational  level. 

It  usually  employs  more  comprehensive  tests  than  either  PM  or  FL. 

FL  tests  are  often  invasive  and  require  that  the  equipment  or  system 
be  taken  off  line  for  fault  localization  and  repair.  In  high  reliability 
systems,  the  adverse  effect  of  off-line  localization  and  repair  is 
minimized  by  the  use  of  (parallel)  redundant  equipment. 
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PM/FD/FL  DESIGN  CONSIDERATIONS 


Ideally,  PM/FD/FL  should  be  run  noninvasively .  As  a  minimum  it  should 
be  run  on  algorithms  that  are  predetermined  not  by  an  operator,  but  rather 
by  priority  needs  of  each  of  the  subsystems  that  comprise  the  total  system. 
The  on-line  portions  of  PM/FD/FL  are  operated  at  a  cyclic  rate  that  is 
constrained  by:  computer  and  system  architecture;  logic  design;  the 
resident  executive  operating  system;  software  language;  and  software  speed 
requirements. 

System  design  should  allow  for  stopping  system  operations  for  an 
advanced  PM/FD/FL  check,  but  the  more  manual  the  method,  the  less  effective 
the  design  philosophy  of  the  PM/FD/FL  functions.  The  major  PM/FD/FL  design 
criteria  is  allowance  for  consistency  in  determination  of  system  integrity 
without  intervention  of  the  operator.  Consistency  and  repeatability  should 
be  given  high  priority  during  design  of  the  PM/FD/FL  functions. 

Performance  monitoring  is  a  macro  measurement  of  the  health  of  the 
system.  It  determines  system  integrity  by  treating  the  system  as  a  "black 
box"  that  responds  to  predetermined  inputs  by  producing  expected  outputs. 

PM  should  not  be  under  operator  control;  be  run  interactively  as  an 
interactive  process;  be  running  at  all  times  when  the  system  is  powered  up 
and  initialized.  PM  should  clearly  inform  the  operator  of  system  status  by 
keying  status  displays  such  as:  "All  monitored  subsystems  and  data  in  the 
PM  functions  are  satisfactory  at  this  time."  As  a  by  product  of 
performance  monitoring,  the  PM  function  may  also  detect  and  localize  faults 
to  some  extent.  The  extent  of  fault  detection  and  localization  depends  on 
the  design  of  the  PM  function.  Problems  detected  by  the  PM  design  are 
called  faults.  They  may  not  be  as  clearly  defined  as  the  FD  type  fault 
detection  system  and  could  cause  confusion  due  to  the  differences  in 
terminology  with  the  faults  detected  by  the  FD  system. 

The  major  purpose  of  the  FD  system  is  to  detect  faults.  It  usually 
accomplishes  this  by  using  what  is  known  as  the  "white  box"  method.  This 
method  bases  fault  detection  on  testing  for  selected  criteria  that  are 
unique  to  specific  modules.  To  the  greatest  extent  possible,  FD  should  be 
performed  automatically  on-line.  It  is  then  free  of  operator  ability  and 
its  integrity  is  determined  by  the  design  philosophy.  In  the  on-line  mode, 
FD  provides  limited  support  of  fault  isolation.  In  some  systems,  FD  tests 
may  also  be  performed  on  a  scheduled  off-line  basis  to  obtain  greater  fault 
detection  sensitivity. 

Fault  localization  should  be  designed  so  that  fault  isolation  at  the 
organizational  level  can  be  quickly  performed  by  maintenance  personnel  with 
minimal  skills,  without  the  use  of  external  equipment,  and  with  minimal 
requirements  for  maintenance  assist  modules. 
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A  fault  is  present  whenever  anything  doesn't  work  as  expected.  It  may 
or  may  not  be  detectable  by  the  FD  function,  and  it  may  or  may  not  result 
in  system  failure.  The  objective  is  to  design  the  FD  function  for  the 
highest  possible  coverage  of  fault  detection,  and  to  clearly  distinguish 
between  faults  that  cause  system  failure  and  those  that  do  not. 

Designers  of  PM/FD  functions  may  find  the  Major,  Minor,  and 
Recoverable  classification  of  faults  to  be  useful.  Major  faults  result  in 
total  loss  of  system  function,  which  cannot  be  recovered  without  off-line 
maintenance  action.  Minor  failures  result  in  performance  degradation,  but 
not  below  allowable  performance  thresholds.  Recoverable  failures  are  those 
which  allow  the  system  to  function  as  intended  without  significant 
interruption  of  performance  after  some  minor  action,  such  as 
reconfiguration  or  reinitialization,  is  taken. 

PM/FD/FL  TAILORING 

The  previous  discussion  pertains  to  a  large,  complex,  and  mission 
essential  system  with  low  tolerance  for  interruption  of  operation  and  with 
high  capability  displays  that  are  manned  continuously.  The  operator  knows 
system  status  through  observation  of  PM  messages,  and,  when  either  an 
unfavorable  PM  message  or  a  fault  message  is  displayed,  he  initiates 
maintenance  actions  and  guides  those  actions  through  observation  and 
reporting  of  the  FL  messages  that  are  given.  In  such  a  system,  the  PM,  FD, 
and  FL  functions  are  almost  equally  important.  There  are  several  other 
types  of  systems  where  this  is  not  so.  Two  types  of  systems  with  different 
PM/FD/FL  requirements  are  discussed  in  the  following  paragraphs.  These 
examples  show  how  important  it  is  to  tailor  the  PM/FD/FL  design  to  the 
requirements . 


A  missile  may  have  a  PM  function  used  for  preventive  maintenance  and  pre¬ 
firing  checks,  but  no  FD  or  FL  functions  that  operate  at  the  organizational 
level.  When  PM  indicates  a  problem,  the  equipment  is  usually  removed  and 
returned  to  the  depot  for  repairs.  In  addition  operation  of  PM  requires  a 
test  set  that  is  external  to  the  weapon. 

An  unmanned  mission-essential  system  may  have  PM,  FD,  and  FL 
functions,  but  then  PM  is  of  much  greater  importance  than  FD  or  FL  because 
there  is  no  operator  to  note  changes  in  performance  through  visual 
indications  and  the  method  of  diagnostics  differs  from  that  of  a  manned 
system.  In  this  case,  automatic  notification  of-changes  in  system  integrity 
is  of  primary  importance.  Such  systems  usually  have  very  high  reliability 
to  minimize  the  probability  of  failure  during  a  mission.  However  the  PM  and 
FD  functions  must  be  capable  of  identifying  or  localizing  faults 
sufficiently  to  allow  repairs  when  the  system  is  retrieved. 
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THE  PM/FD/FL  PROGRAM 
SECTION  II:  REQUIREMENTS 


1 . 0  SCOPE 

1.1  Purpose .  This  report  contains  uniform  procedures  and  methods  for 
developing  and  implementing  a  program  for  the  design,  assessment,  and 
validation  of  Performance  Monitoring,  Fault  Detection,  and  Fault 
Localization  (PM/FD/FL)  functions  within  the  constraints  of  system 
development  programs . 

1.2  Application.  The  procedures  and  methods  of  this  report  are 
applicable  to  the  development  of  PM/FD/FL  functions  for  Department  of 
Defense  electronic  systems  based  on  embedded  computers  and  processors.  The 
methods  and  procedures  of  this  document  are  to  be  applied  as  applicable 
during  the  Conceptual,  Demonstration  and  Validation,  Full  Scale  Engineering 
Development,  and  Production  phases  of  the  system  acquisition  process. 

1.3  Tailoring.  Tasks  described  in  this  report  are  to  be  tailored  as 
appropriate  to  program  phases  and  the  particular  needs  of  the  system  or 
equipment  acquisition  program. 

2 . 0  RELATED  DOCUMENTS 

This  report  is  intended  to  stand  alone,  but  it  is  consistent  with  the 
methods,  procedures,  and  definitions  of  the  documents  referenced  below. 


Military  Standards 

DOD-STD-2 167 

Defense  System  Software  Development 

DOD-STD-2 168 

Defense  System  Software  Quality  Program 

MIL-STD-470 

Maintainability  Program  for  Systems  and 
Equipment 

MIL-STD-47 1 

Maintainability  Verification/  Demonstration/ 
Evaluation 

MIL-STD-721 

Definition  of  Effectiveness  Terms  for 
Reliability  and  Maintainability 

MIL-STD-785 

Reliability  Program  for  Systems  and 

Equipment,  Development  and  Production. 

MIL-STD-1309 

Definition  of  Terms  for  Test,  Measurement  and 
Diagnostic  Equipment 

MIL-STD-1388-1 

Logistic  Support  Analysis 

MIL-STD-2 165 

Testability  Program  for  Electronic  Systems 
and  Equipments. 
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3.0  DEFINITIONS  AND  ACRONYMS 

3.1  Definitions .  Terms  used  in  this  report  are  defined  in 
Appendix  B.  They  are  consistent  with  the  definitions  in  2.0  Related 
Documents . 


3.2  Acronyms .  The  acronyms  used  in  this  document  are  defined 

as  indicated  in  this  paragraph. 


CDR 

CDRL 

D&V 

DID 

EDM 

FD 

FIG 

FL 

FSED 

I  PR 

IV&V 

LRU 

PDR 

PIDS 

PM 

PM/FD/FD 

SOW 


Critical  Design  Review 
Contract  Data  Requirements  List 
Demonstration  and  Validation 
Data  Item  Description 
Engineering  Development  Model 
Fault  Detection 
Fault  Isolation  Group 
Fault  Localization 

Full  Scale  Engineering  Development 
In-Process  Review 

Independent  Verification  and  Validation 
Lowest  Replaceable  Unit 
Preliminary  Design  Review 
Prime  Item  Development  Specification 
Performance  Monitoring 

Performance  Monitoring,  Fault  Detection,  and  Fault 

Localization 

Statement  of  Work 


4 . 0  GENERAL  REQUIREMENTS 

4.1  Scope  of  PM/FD/FL  Program.  The  program  plan  of  this  report 

identifies  inter-disciplinary  efforts  required  to  develop  PM/FD/FL 
functions  that  meet  mission  and  system  requirements.  The  scope  of 
these  efforts  includes: 

1.  Support  of  and  integration  with  maintainability 
design,  including  requirements  for  performance 
monitoring,  fault  detection,  and  fault  localization 
at  the  organizational  level; 

2.  Support  of  integrated  logistic  support  requirements, 
including  minimization  of  requirements  for  spares, 
maintenance  assist  modules,  and  test  equipment? 

3.  Support  of  and  integration  with  design  engineering 
requirements,  including  the  hierarchical  development  of 
PM/FD/FL  requirements  from  the  system  to  the  piece  part; 

4.  Assurance  of  a  balance  between  determination  of  system 
integrity  and  maintenance  assistance  that  meets  the  needs 
of  the  mission  and  equipment  application. 
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4.2  PM/FD/FL  Requirements.  A  combined  PM,  FD,  and  FL  program 
shall  be  established  which  accomplishes  the  following  general 
requirements : 

1.  Preparation  of  a  PM/FD/FL  program  plan; 

2.  Establishment  of  PM/FD/FL  requirements  that  are 
consistent  with  mission  requirements  and  equipment 
application; 

3.  Coordination  of  PM,  FD,  and  FL  software  and  hardware  design 

with  other  software  and  hardware  design 
efforts  to  ensure  integration  of  PM,  FD, 

FL  with  operational  hardware  and  software; 

4.  Quantitative  and  qualitative  evaluation  of  the  extent  to 
which  the  PM,  FD,  and  FL  designs  meet  requirements; 

5.  Inclusion  of  PM,  FD,  FL  in  design  and  program 
review  processes. 

4.3  Application  of  Requirements.  Detailed  requirements  described  in 

this  document  are  to  be  selectively  applied  and  are  intended  to  be 
tailored,  as  required,  and  as  appropriate  to  particular  system  and 
equipment  acquisition  programs.  Appendix  A  provides  rationale  and  guidance 
for  the  selection  and  tailoring  of  PM,  FD,  and  FL  tasks. 

5.0  DETAILED  REQUIREMENTS 

5.1  Program  Elements.  The  major  elements  of  a  full  fledged  PM/FD/FL 

program  and  the  acquisition  phase  in  which  each  element  is  operative  are 

identified  in  Table  1.  Numbered  elements  are  contractor  tasks. 


TABLE  1 

PM/FD/FL  PROGRAM  ELEMENTS 

PROGRAM  ELEMENT  PROGRAM  PHASE 


ELEMENT 

ELEMENT  NAME 

D&v 

FSED 

PROD 

100 

Requirements  Determination 

Program  Surveillance  and  Control 

yes 

yes 

no 

101 

Program  plan 

yes 

yes 

yes 

102 

Monitor/control  subcontractors/vendors 

no 

yes 

yes 

103 

Program  reviews 

yes 

yes 

no 

104 

200 

Configuration  management 

Design  and  Evaluation 

no 

yes 

yes 

201 

Modeling 

yes 

yes 

no 

202 

Allocation 

yes 

yes 

no 

203 

Prediction 

no 

yes 

no 

204 

Fault  tree  analysis 

no 

yes 

no 

205 

Fault  identification  analysis 

no 

yes 

no 

206 

Fault  impact  analysis 

no 

yes 

no 

207 

300 

Transient  smoothing  analysis 

Development  and  Production  Testing 

no 

yes 

no 

301 

Function  certification 

no 

yes 

no 

302 

Independent  verification  &  validation 

no 

yes 

no 

304 

Factory  Acceptance  Test  (FAT) 

no 

no 

yes 

Fleet  Assessment 

no 

yes 

yes 
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5.1.1  Requirements  Determination.  The  Government  program  manager  will 
develop  the  PM,  FD,  and  FL  requirements  and  determine  the  emphasis  on  each 
of  them  during  Demonstration  and  Validation  (D&V) .  Quantitative 
requirements  will  be  determined  through  mission  and  trade-off  analyses. 
These  requirements  will  be  refined  during  the  early  stages  of  Full  Scale 
Engineering  Development  (FSED) . 

5.1.2  Field  Assessment.  The  Engineering  Development  Models  (EDM)  and  the 
first  few  Production  systems  will  be  monitored  by  the  Government  program 
manager  for  signs  of  problems  or  weaknesses  in  the  PM,  FD,  and  FL 
functions.  This  monitoring  will  be  a  subset  of  the  monitoring  that  will  be 
used  to  assess  the  system  for  signs  of  reliability,  maintainability,  and 
logistics  problems. 

5.2  Task  Descriptions.  The  program  plan  portion  of  this  report 
contains  a  comprehensive  set  of  PM/FD/FL  task  descriptions.  These  are 
intended  for  tailored  incorporation  in  each  Statement  of  Work  (SOW)  of 
acquisition  contracts  for  Department  of  Defense  (DOD)  electronic  systems 
and  equipment.  In  this  report,  the  acronyms  PM , FD ,  and  FL  rather  than 
PM/FD/FL  are  used  whenever  it  is  necessary  to  emphasize  the  fact  that  the 
three  functions  can  receive  different  degrees  of  attention  depending  on 
mission  requirements  and  equipment  application.  The  PM/FD/FL  tasks 
intended  for  incorporation  in  appropriate  SOWs  are  identified  in  Table  2. 

TABLE  2 


PM/FD/FL  TASK  REQUIREMENTS  (CONTRACTOR) 

TASK 

FUNCTION 

APPLICATION 

NUMBER 

TASK  NAME 

PM 

FD 

FL 

100 

101 

Program  Surveillance  and  Control 

Program  plan 

yes 

yes 

yes 

102 

Monitor/control  subcontractors/vendors 

yes 

yes 

yes 

103 

Program  reviews 

yes 

yes 

yes 

104 

Configuration  management 

yes 

yes 

yes 

200 

201 

Design  and  Evaluation 

Modeling 

yes 

yes 

yes 

202 

Allocation 

yes 

yes 

yes 

203 

Prediction 

yes 

yes 

yes 

204 

Fault  tree  analysis 

yes 

yes 

no 

205 

Fault  identification  analysis 

no 

yes 

yes 

206 

Fault  impact  analysis 

yes 

yes 

r.o 

207 

Transient  smoothing  analysis 

no 

yes 

no 

300 

301 

Development  and  Production  Testing 

Function  certification 

yes 

yes 

yes 

302 

Independent  verification  &  validation 

yes 

yes 

yes 

304 

Factory  Acceptance  Test  (FAT) 

no 

no 

yes 
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5.2.1  Structuring  the  Program.  Development  of  the  PM/FD/FL  functions  is 
a  subset  of  the  system  development  program.  The  resulting  commonality  of 
development  methodology  permits  several  PM/FD/FL  tasks  to  be  integrated 
into  system  tasks,  but  care  must  be  taken  to  ensure  that  visibility  into 
PM/FD/FL  remains  after  integration  of  tasks.  There  must  be  sufficient 
visibility  into  PM/FD/FL  development  to  ensure  that  its  development  occurs 
simultaneously  with,  and  retains  the  same  momentum  as,  the  development  of 
other  functions.  In  addition,  visibility  must  be  sufficient  to  ensure  early 
identification  of  problems  and  confidence  in  achievement  of  requirements. 
The  PM/FD/FL  program  must  be  structured  to  ensure  this  visibility  while 
optimizing  the  use  of  development  resources  and  eliminating  duplication  of 
effort. 


The  PM,  FD,  and  FL  programs  must  also  be  structured  to  consider  the 
impact  of  policy  documents  such  as  DOD  4245. 7-M  (Transition  from 
Development  to  Production  -  Solving  the  Risk  Equation) .  Such  documents 
will  be  identified  in  the  Applicable  Documents  section  of  the  contract 
Statement  of  Work. 

Tasks  and  program  elements  will  be  selected  and  tailored  to  match 
the  emphasis  placed  on  each  of  the  PM/FD/FL  functions  and  to  match  the 
acquisition  phase  that  is  being  addressed.  For  example,  the  greatest 
emphasis  may  be  placed  on  PM  (system  integrity)  in  the  case  of  a  mission 
critical  unmanned  system,  while  FL  may  receive  the  greatest  emphasis  in  the 
case  of  a  mission  critical  system  that  is  continuously  manned. 

5.2.2  Program  Interfaces.  The  PM,  FD,  and  FL  program  interfaces  with: 
system  development,  design,  test,  and  demonstration;  Configuration 
Management;  Reliability;  Maintainability;  Integrated  Logistics  Support; 
Safety;  and  Human  Factors.  During  development  and  assessment,  essential 
data  is  sent  to,  and  received  from,  these  and  other  disciplines  through  the 
interfaces.  It  is  important  that  these  interfaces  be  recognized  by  those 
managing  the  PM,  FD,  and  FL  programs,  and  that  these  communications  be  kept 
open. 

6 . 0  DATA  REQUIREMENTS 

6.1  Deliverable  Data.  When  this  report  is  used  in  an  acquisition,  the 
data  identified  in  this  paragraph  shall  be  deliverable  only  when  specified 
on  the  DD  Form  1423  Contract  Data  Requirement  List  (CDRL) .  When  the  DD 
Form  1423  is  not  used  and  the  Defense  Acquisition  Regulation  7-104.9 (n)  is 
cited,  the  data  identified  below  shall  be  delivered  in  accordance  with  the 
requirements  specified  in  the  contract  or  purchase  order.  Deliverable  data 
associated  with  the  requirements  of  this  document  are  identified  in  the 
Data  Item  Descriptions  (DID)  of  Table  3.  Data  item  descriptions  for  related 
requirements  are  also  included. 

6 . 2  Document  Samples. 

Samples  of  contract  SOWs  and  CDRL  items  are  provided  in  Appendices 
C  and  D,  respectively. 
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TASK 


100 

101 

103 

104 

200 

201 

202 

203 

204 

205 

206 
207 

300 

301 

302 
304 


Additional 


TABLE  3 

DATA  REQUIREMENTS 


DATA  REQUIREMENT 


APPLICABLE  DATA  ITEM 
DESCRIPTION  (DID) 


PROGRAM  SURVEILLANCE  AND  CONTROL 
Program  plans 
Design  reviews 
Configuration  management 


DI-ATTS-80005 
DI-E  5423 
DI-E-3108 


DESIGN  AND  EVALUATION 

Modeling 

Allocation 

Prediction 

Fault  tree  analysis 

Fault  identification  analysis 

Fault  impact  analysis 

Transient  smoothing  analysis 


DI-MNTY-80825 
DI— MNTY-80826 
DI-MNTY-80827 
DI-MISC-80048 
DI-MISC-80048 
DI-MISC-80048 
DI-MISC-80048 


DEVELOPMENT  AND  PRODUCTION  TESTING 
Function  certification 
Independent  verification  &  validation 
Factory  Acceptance  Test  (FAT) 


UDI-T-23732B 

UDI-T-23732B 

UDI-T-23732B 


Data  Item  Descriptions: 

DATA  COLLECTION 
TESTABILITY  PROGRAM  PLAN 
TESTABILITY  ANALYSIS  REPORT 


DI—MNTY— 80824 
DI-T-7198 
DI— T— 7199 
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PERFORMANCE  MONITORING,  FAULT  DETECTION,  FAULT  LOCALIZATION  DESIGN  GUIDANCE 


THE  PM/FD/FL  PROGRAM 
SECTION  III  -  TASK  DESCRIPTIONS 


INTRODUCTION 

In  this  section,  the  usual  PM/FD/FL  acronym  has  been  changed  to  PM, 

FD,  and  FL  to  emphasize  the  semi-independence  of  these  functions.  This 
semi-independence  is  needed  because  the  emphasis  placed  on  each  of  these 
functions  will  vary  with  the  application. 

To  provide  complete  visibility  into  the  PM,  FD,  and  FL  development 
process,  each  task  of  this  section  is  fully  described.  In  practice,  several 
of  the  tasks  may  be  combined  with  similar  tasks  of  other  disciplines 
provided  that  visibility  into  the  PM,  FD,  and  FL  development  process  is  not 
lost.  PM,  FD,  and  FL  candidates  for  combination  with  tasks  of  other 
disciplines  are: 

o  Program  Plan 

o  Control  of  Subcontractors  and  Suppliers 
o  Design  Reviews 
o  Configuration  Management 
o  Independent  Verification  and  Validation 
o  Function  certification 

All  other  tasks  of  this  section  are  unique  to  PM,  FD,  and/or  FL. 

The  contractor  shall  provide  justification,  and  receive  approval  of  the  DOD 
Program  Manager,  before  combining  tasks. 

The  major  task  groupings  of  this  section  are: 

o  Program  surveillance  and  control ; 
o  Design  and  evaluation; 
o  Development  and  production  testing. 

All  of  the  tasks  of  this  section  are  to  be  performed  by  the  contractor. 

The  DOD  program  manager  has  review  and  approval  authority  over  the  work 
performed  under  these  tasks. 

It  is  important  that  the  contractual  requirements  of  each  task  be 
tailored  to  system  requirements  and  fully  described  in  the  SOW  of  the 
contract.  The  DIDs  used  to  get  deliverable  documents  are  adaptations  of 
existing  DIDs  that  do  not  fully  describe  PM/FD/FL  requirements.  New  DIDs, 
specifically  designed  for  PM/FD/FL  reporting,  will  be  required  in  the 
future. 


11. 


TASK  101 


PM,  FD,  AND  FL  PROGRAM  PLAN 


101.1  Overview.  The  PM,  FD,  and  FL  Program  Plan  (Plan)  shall  be 
designed  as  a  basic  tool  to  assist  the  contractor  in  implementing  an 
effective  PM,  FD,  and  FL  development  program.  The  Government  Program 
Manager  will  use  the  Plan  to  (1)  evaluate  the  contractor's  approach  to,  and 
his  execution  of,  PM,  FD,  and  FL  tasks,  (2)  evaluate  the  adequacy  of  his 
procedures  for  planning,  implementing,  and  controlling  the  PM,  FD,  and  FL 
tasks,  and  (3)  evaluate  the  ability  of  his  organizational  structure  to 
focus  on  PM,  FD,  and  FL  activities/problems. 

101.2  Purpose.  The  purpose  of  Task  101  is  to  develop  a  PM,  FD, 
and  FL  Program  Plan  that  identifies  and  integrates  all  program  tasks 
necessary  to  accomplish  the  requirements  of  the  Prime  Item  Development 
Specification  (PIDS)  and  the  Statement  of  Work  (SOW) . 

101.3  Task  Description.  A  PM,  FD,  and  FL  Program  Plan  shall  be 
prepared  to  provide,  as  a  minimum,  the  following: 

1.  A  description  of  how  the  program  will  be  conducted  to  meet 
the  requirements  of  the  PIDS  and  the  SOW. 

2.  A  description  of  how  PM,  FD,  and  FL  designs  interface  with 
total  system  design. 

3.  A  detailed  description  of  how  each  specific  PM,  FD,  and  FL 
requirement  will  be  performed  or  complied  with. 

4 .  The  procedures  to  evaluate  the  status  and  control  of  each 
task,  and  identification  of  the  organizational  unit  with 
the  authority  and  responsibility  for  executing  each  task. 

5.  Description  of  the  management  structure,  including 
interrelationship  between  line,  service,  staff,  and  policy 
organizations. 

6.  Identification  of  key  personnel  for  managing  the  PM,  FD, 
and  FL  program  and  the  level  of  authority  for  problem 
resolution. 

7.  The  method  by  which  the  PM,  FD,  and  FL  requirements  are 
disseminated  to  designers  and  associated  personnel,  and  how 
design  interfaces  are  accomplished. 

8.  A  schedule  with  estimated  start  and  completion  points  for 
each  PM,  FD,  and  FL  program  activity  or  task. 


9.  The  designation  of  PM,  FD,  and  FL  milestones,  including 
design,  design  review  (PDR,  CDR,  IPR) ,  and  test. 

10.  The  identification  of  known  PM,  FD,  and  FL  problems  to  be 
solved,  an  assessment  of  the  impact  of  these  problems  on 
meeting  specified  requirements,  and  the  proposed  solutions 
or  proposed  plan  to  solve  them. 

11.  The  procedure  or  methods  for  recording  the  status  of 
actions  taken  to  resolve  problems. 

When  approved  by  the  Government  Program  Manager,  the  PM,  FD,  and  FL  Program 
Plan  shall  become  a  basis  for  evaluation  of  contractual  compliance. 

TASK  102  MONITOR/CONTROL  OF  SUBCONTRACTORS  AND  SUPPLIERS 

102.1  Overview.  Monitoring  and  control  of  subcontractors  and  suppliers 
provides  visibility  into  subcontractor/ supplier  achievement  of  flowed  down 
PM,  FD,  and  FL  requirements  and  the  control  needed  to  ensure  achievement  of 
requirements.  Monitoring  and  control  also  ensures  that  subcontractor 
/supplier  PM,  FD,  and  FL  efforts  remain  consistent  with  system  PM,  FD,  and 
FL  design  and  requirements. 

102.2  Purpose.  The  purpose  of  Task  102  is  to  provide  the  prime 
contractor  with  appropriate  surveillance  and  management  control  of 
subcontractor  and  supplier  PM,  FD,  and  FL  efforts  so  that  timely  management 
action  can  be  taken  as  the  need  arises. 

102.3  Task  Description .  The  contractor  shall  insure  that  system 
elements  obtained  from  subcontractors  and  suppliers  will  meet  the  flowed 
down  PM,  FD,  and  FL  requirements.  All  subcontracts  shall  include 
provisions  for  review  and  evaluation  of  the  subcontractors'  PM,  FD,  and  FL 
efforts  by  the  prime  contractor,  and  by  the  procuring  activity  at  its 
discretion.  The  contractor  shall,  as  appropriate: 

1.  Incorporate  quantitative  PM,  FD,  and  FL  requirements  in 
subcontracted  equipment  specifications. 

2.  Assure  that  subcontractors  have  a  PM,  FD,  and  FL  program  that  is 
consistent  with  the  system  PM,  FD,  and  FL  program  and  includes 
provisions  to  review  and  evaluate  their  suppliers'  FD  and  FL 
efforts. 

3.  Attend  and  participate  in  subcontractors'  design  reviews. 

4.  Review  subcontractors'  PM,  FD,  and  FL  analyses  for  accuracy 
and  correctness  of  approach. 

5.  Review  subcontractors'  test  plans,  procedures,  and  reports 
for  accuracy  and  correctness  of  approach. 


13. 


TASK  103  PM,  FD,  AND  FL  DESIGN  REVIEWS 

103.1  Overview.  Design  reviews  shall  be  held  to  determine  whether  or 
not  the  projected  PM,  FD,  and  FL  designs  will  meet  the  requirements  of  the 
specifications  and  to  assess  the  suitability  of  hardware  and  software 
design.  At  the  onset,  reviews  should  be  held  frequently  to  ensure  that  the 
contractor  does  not  proceed  with  unsuitable  designs.  The  reviews  should 
also  confirm  that  the  contractor  is  meeting  the  intent  as  well  as  the 
wording  of  the  specification.  It  is  not  necessary  that  the  PM,  FD,  and  FL 
be  separate  from  other  reviews,  providing  that  PM,  FD,  and  FL  receive  the 
visibility  needed  to  ensure  that  the  contractor  is  performing  and  adhering 
to  specifications  and  contract  requirements. 

103.2  Purpose.  The  purpose  of  Task  103  is  to  establish  a  requirement 
for  the  contractor  to  conduct  formal  and  informal  PM,  FD,  and  FL  design 
reviews. 

103.3  Task  Description.  PM,  FD,  and  FL  formal  design  reviews  shall  be 
conducted  in  accordance  with  the  requirements  of  MIL-STD-1521B  or  a 
schedule  approved  by  the  Government.  Informal  in-process  PM,  FD,  and  FL 
reviews  shall  be  conducted  at  least  quarterly  with  a  mutually  agreed  upon 
schedule  by  the  Government  and  Contractor  until  formal  Critical  Design 
Reviews  are  intiated.  In  addition  to  the  design  review  requirements  of  MIL- 
STD-1521B,  the  following  design  reviews  shall  include  review  of  the  PM,  FD, 
and  FL  items  indicated  below. 

1.  Preliminary  Design  Review  (PDR) : 

a.  Updated  PM,  FD,  and  FL  program  status  including: 

1)  PM,  FD,  FL  modeling; 

2)  PM,  FD,  FL  allocation; 

3)  PM,  FD,  FL  predictions; 

4)  PM,  FD,  FL  specification  compliance/traceability; 

5)  Design  guideline  criteria. 

b.  Problems  affecting  PM,  FD,  and/or  FL. 

c.  PM,  FD,  and/or  FL  critical  items. 

2.  Critical  Design  Review  (CDR) : 

a.  PM,  FD,  and  FL  compliance  with  specifications. 

b.  PM,  FD,  and  FL  predictions  and  analyses. 

c.  PM,  FD,  and  FL  critical  items. 

d.  Problems  affecting  PM,  FD,  and/or  FL. 

e.  Identification  of  functions  requiring  high  reliability 

or  a  large  number  of  lines  of  code  software/ firmware. 

f.  Analysis  results. 
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In-Process  PM,  FD,  and  FL  Reviews  (IPR) : 


a.  Consideration  of  those  PM,  FD,  and  FL  items  identified 
in  subparagraph  1  and  2 . 

b.  Results  of  PM,  FD,  and  FL  tests  and  analyses. 

c.  Test  schedule:  start  and  completion  dates. 

d.  PM,  FD,  and/or  FL  parts,  design,  reliability,  and 
schedule  problems. 

e.  Status  of  PM,  FD,  and/or  FL  action  items. 

f.  Contractor's  assessment  of  PM,  FD,  and  FL  design 
effectiveness . 

g.  Other  topics  and  issues  as  needed. 

h.  Results  of  applicable  PM,  FD,  and  FL  testing. 

4.  Test  Readiness  Review: 

a.  PM,  FD,  and  FL  analyses  status  and  PM,  FD,  and  FL 
predictions. 

b.  Test  schedule. 

c.  Test  profile. 

d.  Test  plan  including  failure  definitions. 

e.  Test  report. 

5.  Production  Readiness  Review:  Results  of  applicable  PM,  FD, 
and  FL  testing. 

TASK  104  PM,  FD,  AND  FL  CONFIGURATION  MANAGEMENT 

104.1  Overview.  The  contractor  shall  develop  and  implement 
Configuration  Management  procedures  which  provide  technical  and 
administrative  direction  and  surveillance  to: 

1.  Identify  and  document  the  functional  and  physical 
characteristics  of  each  hardware  and  software/ firmware 
configuration  items  of  the  PM,  FD,  and  FL  functions. 

2.  Control  changes  to  these  characteristics. 

3.  Record  and  report  the  processing  of  changes  and  the 
status  of  implementation.  The  contractor  shall  perform 
PM,  FD,  and  FL  CM  within  the  framework  of  the  system 
CM. 

104.2  Purpose .  The  purpose  of  this  task  is  to  provide  the  contractor 
and  the  Government  Program  Manager  with  the  information  needed  to  identify 
the  initial  hardware  software/ firmware  configuration  of  the  PM,  FD,  and  FL 
functions  and  to  track  the  status  and  effects  of  change-proposal  and  change 
implementation  actions . 

104.3  Task  Description.  The  contractor  shall  apply  CM  to  the  hardware, 
software,  and  firmware  of  the  PM,  FD,  and  FL  function  within  the  framework 
of  the  hardware  and  software  CM  of  the  system.  System  and  function  CM  shall 
be  in  accordance  with  MIL-STD-483,  MIL-STD-490  and  DOD-STD-2167  as  tailored 
by  the  contract  and  government  requirements  document.  Data  pertaining  to 
the  PM,  FD,and  FL  shall  include,  but  not  be  limited  to: 
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1.  Requirements  as  provided  to  subcontractors. 

2.  Subcontractors  response  and  interpretation  of  requirements. 

3 .  Test  procedures  by  contractor  and  subcontractors . 

4.  Any  qualification  tests,  results,  conclusions,  and/or 
observations. 

5.  Changes  as  provided  by  the  program  office,  as  initiated  by  the 
contractor,  as  required  by  the  results  from  new  data  as  required 
for  any  other  purposes. 

6.  All  data  item  requirements. 

7.  All  data  necessary  for  life  cycle  support  and  test 
certification. 

8.  Design  drawings,  source  code,  program  language ( s) ,  and  other 
documentation  to  provide  the  capability  for  independent 
certification,  duplication  of  the  system,  subsystem,  elements, 
firmware/software,  and  hardware. 

TASK  201  PM,  FD,  AND  FL  MODELING 

201.1  Overview.  Both  quantitative  and  qualitative  analyses  are  useful 
in  determining  where  PM,  FD,  and  FL  resources  should  be  applied.  The 
analyses  identify  design  and  quality  improvements  that  must  be  made  if 
requirements  are  to  be  met.  In  particular,  the  analyses  are  efficient  work 
direction  tools  because  they  can  confirm  system  adequacy  or  identify  the 
need  for  design  change,  provided  they  are  accomplished  in  conjunction  with, 
or  reviewed  by,  other  disciplines.  Models  provide  the  basis  for  assessment 
of  PM,  FD,  and  FL  performance,  effectiveness,  and  system  impact.  They  are 
used  in  the  allocation  of  system  level  requirements  to  specific  hardware 
and  software  functions,  and  in  the  prediction  of  performance  parameters. 

The  PM,  FD,  and  FL  models  are  derived  from  the  system  reliability  and 
maintainability  models. 

201.2  Purpose.  The  purpose  of  Task  201  is  to  develop  PM,  FD, 

and  FL  models  for  making  numerical  allocations  and  estimates  to  evaluate 
system/subsystem/equipment  PM,  FD,  and  FL  monitoring  effectiveness. 

201.3  Task  Description.  PM,  FD,  and  FL  mathematical  models  based  on 
system/ subsystem/ equipment  functions  shall  be  developed  and  maintained.  As 
the  design  evolves,  PM,  FD,  and  FL  block  diagram  (Fault  Isolation 
Groupings)  (FIGs)  with  associated  allocations  and  predictions  for  all 
elements  in  the  FIG  shall  be  created.  The  PM,  FD,  and  FL  block  diagrams 
shall  be  keyed  and  traceable  to  the  functional  block  diagram,  schematics, 
drawings,  and  specifications. 
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The  model  outputs  shall  be  expressed  in  terms  of  PM,  FD,  and  FL 
requirements.  As  changes  occur,  the  model  shall  be  updated  to 
include  hardware  and/or  software/ firmware  design  changes.  The  PM,  FD, 
and  FL  models  shall  be  updated  with  information  resulting  from 
relevant  tests  and  changes  in  item  configuration. 

TASK  202  PM,  FD,  AND  FL  ALLOCATION 

202.1  Overview.  System  PM,  FD,  and  FL  requirements  evolve  in  a  number 
of  ways  from  informed  judgments  to  analyses  based  on  empirical  data.  The 
requirements  are  designed  to  minimize  the  total  cost  of  developing, 
procuring,  and  operating  tne  system  during  its  life  cycle.  The  integrity 
of  the  system  is  maintained  by  adequate  top-down  design  that  ensures  the 
ability  of  the  system  to  meet  specified  requirements.  Allocated 
requirements  must  be  iteratively  refined  before  resources  can  be 
specifically  designated  to  meet  the  requirements. 

202.2  Purpose .  The  purpose  of  Task  202  is  to  ensure  that,  once 
quantitative  system  requirements  have  been  determined,  they  are  allocated 
or  apportioned  to  lower  levels. 

202.3  Task  Description.  Both  the  mission  and  mission  integrity 
requirements  shall  be  allocated  to  the  level  specified  and  shall  be  used  to 
establish  the  baseline  requirements  for  equipment  and  software/ firmware 
designer.  Requirements  consistent  with  the  top  level  allocations  shall  be 
imposed  on  all  subcontractors  and  suppliers.  The  allocated  values  shall  be 
included  in  appropriate  sections  of  any  procurement  specifications, 
critical  item  specifications,  and  contract  end  item  specifications  to 
subcontract ors/suppl iers . 

All  allocated  PM,  FD,  and  FL  values  established  by  the  contractor 
and  included  in  subcontract  item  specifications  shall  be  consistent 
with  the  mathematical  model  required  in  Task  201. 

TASK  203  PM,  FD,  AND  FL  PREDICTIONS 

203.1  Overview.  Allocations  are  determined  from  the  system  PM,  FD,  and 
FL  requirements  to  provide  lower  level  requirements  which  are  levied  on  the 
designers  and  software/firmware  engineers.  As  design  work  progresses, 
predictions  based  on  previously  generated  data  and  assessments  based  on 
program  test  data  are  used  to  determine  whether  the  allocated  requirements 
can  or  will  be  met. 

Predictions  combine  lower  level  PM,  FD,  and  FL  data  to  indicate  equipment 
PM,  FD,  and  FL  performance  at  successively  higher  levels,  from 
subassemblies  through  subsystem  to  system.  Predictions  falling  short  of 
requirements  at  any  level  signal  the  need  for  management  and  technical 
attention. 
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203.2  Purpose.  The  purpose  of  Task  203  is  to  estimate  PM,  FD,  and  FL 
capabilities  of  the  system,  subsystem,  equipment,  hardware, and 
software/ firmware  and  to  determine  whether  or  not  the  PM,  FD,  and  FL 
requirements  can  be  achieved  with  the  proposed  design. 

203.3  Task  Description.  PM,  FD,  and  FL  predictions  shall  be  made  for 
the  system,  subsystem,  equipment,  hardware,  and  software/firmware.  PM 
predictions  shall  include  the  probability  of:  functional  failure;  not 
diagnosing  a  performance  fault,  and  incorrectly  diagnosing  a  performance 
fault.  FD  parameters  of  interest  are  the  probability  of:  not  diagnosing  a 
fault;  and  incorrectly  diagnosing  a  fault.  FL  parameters  of  interest  are 
the  probability  of:  not  localizing  a  fault;  isolating  a  fault  to  the 
incorrect  Fault  Isolation  Group;  and  localizing  a  fault  to  within  the 
correct  fault  group.  Predictions  shall  be  made  (1)  to  show  the  ability  of 
the  PM,  FD,  and  FL  function  to  assess  system  and  subsystem  integrity,  (2) 
to  provide  a  basis  for  life-cycle  and  logistic  support  analyses,  and  (3)  to 
provide  a  basis  for  estimating  system  availability. 

The  predictions  shall  use  the  associated  PM,  FD,  and  FL  block 
diagrams  of  Task  201  and  PM,  FD,  and  FL  coverage  data  and  shall  be 
approved  by  the  Government.  Items  and  equipment  shall  not  be 
excluded  from  the  prediction. 

TASK  204  PERFORMANCE  MONITORING/FAULT  DETECTION  FAULT  TREE 

204.1  Overview.  The  PM/FD  Fault  tree  is  used  as  a  basic  tool  by  the 
contractor,  the  government  program  office,  and  the  independent  verification 
and  validation  (IV&V)  groups  to  determine  the  path  of  initial  fault 
observation  to  the  final  display. 

204.2  Purpose.  The  specific  purpose  of  the  PM/FD  fault  tree  is  to 
assist  in  designing,  testing,  and  implementing  an  effective  PM/FD. 

The  PM/FD  fault  tree  shall  be  used  to  evaluate  the  contractor's 
approach  to,  and  confirmation  of,  adherence  to  PIDS  requirements. 

204.3  Task  Description.  The  fault  tree  shall  identify  each 
fault  test  point  and  its  pass/fail  levels.  Each  functional  failure 
shall  be  labeled  and  described.  The  description  shall  include: 

1.  All  test  points  that  are  used  to  determine  if  a  functional 
failure  exists.  Where  a  votive  or  count  determination 
(e.g.,  3  out  of  5)  exists,  descriptions  shall  be  supplied. 

2.  Identification  of  test  points  that  are  common  to  any  other 
PM/FD/FL  subprograms  or  tests. 

3.  The  contractor's  verification  that  determinations  of  PM/FD 
fault  indications  are  direct,  not  made  by  inference  or 
other  indirect  observations. 

4.  Proof  that  software/ firmware  programs  that  are  used  for 
determination  are  labeled  and  referenced  to  the 
configuration  item  where  they  are  located. 
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TASK  205  FD  AND  FL  FAULT  IDENTIFICATION  ANALYSIS 

205.1  Overview.  Faults  that  are  detected  roust  also  be  correctly 
identified"!  In  order  to  perform  repair  actions,  much  detail  about  each 
fault  is  required.  The  particular  off-line  tests  using  the  FL  function 
which  identify  the  correct  Fault  Isolation  Group  (FIG)  and  possibly  the 
failing  Lowest  Replacement  Unit  (LRU)  often  require  that  more  than  one  FL 
test  be  performed.  For  this  reason,  all  monitored  test  points  that  provide 
fault  information  to  the  central  PM/FD/FL  function  must  be  correctly 
designed.  The  information  from  these  test  points  must  be  recorded  and 
assimilated  into  proper  groupings,  which  identify  the  suitable  FL  test  to 
be  performed. 

Many  faults  cause  domino  effects  where  the  occurrence  of  one  fault 
causes  additional  fault  indications.  In  order  to  provide  effective  repair, 
in  minimum  time,  and  also  to  evaluate  the  impact  on  the  system  caused  by 
the  root  fault,  it  is  necessary  that  the  root  fault  be  determined  and 
found.  'T’he  design  of  the  fault  localization  subsystem  must  be  of 
sufficient  complexity  to  isolate  the  root  fault  despite  the  occurrence  of 
multiple  faults  and  other  ambiguities. 

205.2  Purpose.  The  purpose  of  Task  205  is  to  verify  that  proper  fault 
identification,  display,  and  maintenance  action  codes  will  be  available  to 
maintenance  personnel.  Verification  shall  also  demonstrate  that  the 
identity  of  any  faults  detected  will  be  prioritized  so  that  maintenance 
personnel  will  perform  tests  for  the  more  likely  fault  first.  Verification 
shall  show  that  the  correct  information  specified  in  the  PIDS  for  each 
detected  fault  is  correctly  provided  to  and  displayed  on  the  maintenance 
panel . 

205.3  Task  Description.  A  fault  identification  plan  for  FD  and  FL  shall 
be  developed  and  include,  but  not  be  limited  to: 

1.  A  description  of  how  fault  identification  is  handled  by  the 
system. 

2  A  description  of  how  the  fault  identification  design  meets 
PIDS  requirements. 

3.  A  test  plan  and  procedure  for  proper  fault  identification. 

4.  A  worst-case  series  of  tests  to  show  that  the  most  likely 
fault  is  displayed  first. 

5.  Test  cases  intended  to  be  ambiguous  with  respect  to  which 
fault  initiated  the  problem. 


19. 


6.  Stress  tests  for  proper  fault  identification  under  actual 
operating  conditions. 

7.  A  listing  of  the  test  panel  indications  for  all  above 
tests.  Documentation  of  all  fault  localization  results 
shall  be  included  in  the  task. 

Note  that  the  fault  identification  test  does  not  apply  to  PM. 

TASK  206  FAULT  IMPACT  ANALYSIS  (PM  AND  FD) 

206.1  Overview.  Not  all  faults  have  the  same  effect  on  system 
integrity‘s  effectiveness,  or  operational  availability.  Some  faults  mask 
others  that  may  have  more  of  an  impact  on  system  integrity.  Similarly, 
certain  portions  of  systems  have  redundancies,  either  natural  or  planned. 

In  the  case  of  faults  in  redundant  portions,  it  may  be  possible  to  schedule 
maintenance  for  some  planned  time.  The  faults,  then,  are  not  critical  to 
system  integrity  or  operations,  provided  they  are  recorded  and  repaired  at 
the  next  repair  cycle  time.  When  a  multitude  of  faults  occur,  there  are 
often  one  or  two  major  faults  that  have  had  a  ripple  effect  and  cause  other 
faults  to  be  displayed.  The  ripple  impact  is  potentially  dangerous  because 
the  impact  on  system  operation  will  not  be  easily  determined  and  the  parent 
fault(s)  of  the  problem  may  not  be  identified.  By  assigning  levels  of 
impact  to  each  fault,  there  is  a  better  probability  of  correctly  assessing 
the  fault  impact,  determining  system  impact,  and  looking  for  the  most 
damaging  fault  first. 

In  effect,  giving  a  level  of  impact  to  each  fault  allows  for  more  correct 
diagnosis  of  the  actual  cause  of  failures.  For  example,  if  a  power  supply 
were  to  be  in  fault,  most  of  the  items  that  had  test  points  for  the 
performance  monitoring  subsystem  would  give  indications  of  failure.  For 
this  reason,  given  the  multitude  of  possible  faults  occurring  or  seeming  to 
occur  all  at  once,  it  is  necessary  to  determine  the  impact  of  every  test 
point  used  for  the  performance  monitoring  subsystem.  The  standard 
procedure  is  to  give  each  fault  an  impact  level  (sometimes  called  a 
priority  level) . 

206.2  Purpose .  The  PM  purpose  of  Task  206  is  to  test  the  ability  of  the 
performance  monitoring  subsystem  to  correctly  determine  the  impact  of 
faults  that  it  has  detected  with  respect  to  the  integrity  and  effectiveness 
of  the  major  system.  The  FD  purpose  of  Task  206  is  to  test  the  ability  of 
the  fault  detection  subsystem  to  correctly  identify  faults  it  has  detected 
and  to  correctly  determine  the  impact  of  those  faults  with  respect  to  the 
integrity  and  effectiveness  of  the  major  system.  Additionally,  this  task 
is  to  demonstrate  that  faults  do  not  mask  each  other  when  they  occur  at  the 
same  time.  This  task  is  also  to  demonstrate  that  the  fault  determination 
will  allow  for  maintenance  actions  in  the  required  time  and  to  the  proper 
FIG. 
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206.3  Task  Description.  A  PM/FD  plan  for  fault  impact  shall  be 
developed  and  include,  but  not  be  limited  to: 

1.  A  description  of  how  fault  impact  is  handled  by  the 
system. 

2.  A  description  of  how  the  PM  meets  the  PIDS  requirements. 

3.  A  test  plan  and  procedure  for  testing  fault  impact. 

4.  A  worst-case  series  of  tests  and  their  evaluations 
regarding  which  fault  created  the  problem. 

5.  Test  cases  intended  to  be  ambiguous  with  respect  to  which 
fault  initiated  the  problem. 

6.  Stress  test  cases  under  actual  operating  conditions. 

7.  A  listing  of  test  panel  indicators  for  all  tests. 

Documentation  of  all  fault  impact  test  results  shall  be  included  in  this 
task. 

TASK  207  FAULT  DETECTION  FUNCTION  TRANSIENT  SMOOTHING  (FD  ONLY) 

207.1  Overview.  Electronic  systems,  especially  those  that  have  long 
distances  between  units,  are  susceptible  to  all  kinds  of  interference, 
including  DC  offsets,  ground  loops,  EMI,  and  noise  bursts  and  pulses  caused 
by  other  electronic  devices.  The  devices  th  selves  may  also  cause 
transients  when  certain  combinations  of  operations  are  performed. 

Therefore,  a  simple  pass/ fail  test  at  any  test  point  may  show  indication  of 
a  fault  when,  in  fact,  there  is  none.  similarly,  a  fault  finding  may  be 
lost  or  erroneously  modified  during  transmission  from  one  system  component 
to  another.  Transient  smoothing  is.  therefore,  required  to  reduce  the 
number  of  false  fault  indications.  It  is  also  imperative  that  certain  test 
points  which  are  critical  to  system  integrity  have  their  responses  quickly 
read.  All  test  points  should  be  able  to  report  within  given  latency  times 
even  if  anomalies  exist  somewhere  in  the  subsystem. 

207.2  Purpose.  The  purpose  of  Task  207  is  to  ensure  the  ability  of  the 
FD  subsystem  to: 

1.  Report  all  faults  within  the  specified  latency  time, 
regardless  of  anomalies,  either  at  the  test  point  or  during 
transmission  from  one  point  to  another. 

2.  Report  the  condition  of  any  test  point  that  has  become 
inoperative  or  incommunicative. 

3.  Not  report  non-recurring  faults,  glitches,  or  transients. 
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207.3  Task  Description.  A  Fault  Detection  Transient  Smoothing  Plan  for 
design,  test,  certification,  and  verification  shall  be  developed  and 
implemented.  The  plan  shall  include,  but  not  be  limited  to: 

1.  A  description  of  how  each  fault  is  handled  to  avoid  false 
alarms . 

2.  A  description  of  verification/validation  test  plans  for 
transient  smoothing. 

3.  A  description  of  verification  of  tests  to  be  performed 
under  worst-case  actual  operating  conditions  or  equivalent. 

4.  A  description  of  the  verification  test  that  ensures  the 
reporting  of  faults  within  the  time  specified  in  the  PIDS. 

A  report  on  the  implementation  of  this  plan,  including  test  findings,  shall 
be  included  in  all  design  reviews. 

TASK  301  FUNCTION  DESIGN  QUALIFICATION  (PM,  FD,  AND  FL) 

301.1  Overview.  It  is  vital  that  designs  be  tested  not  only  to  see  if 
the  designs  themselves  are  functional  and  fault  free,  but  also  that  the 
designs  meet  both  the  intent  as  well  as  the  'letter'  of  the  specification. 
Verification  of  design  to  specification  should  be  accomplished  at  all 
levels  of  development.  When  it  appears  to  have  been  completed,  retesting 
and  reverification  should  occur,  using  the  original  design  teams, 
contractor  quality  assurance  personnel,  independent  test  teams,  and 
finally,  representation  of  the  Government  program  management. 

301.2  Purpose .  The  purpose  of  this  task  is  to  verify  and  demonstrate 
that  the  designs  for  the  PM,  FD,  and  FL  functions  meet  and  show  adherence 
to  PIDS  requirements  in  enough  detail,  quality,  frequency,  and  number  to 
provide  a  high  level  of  confidence  and  achievement  of  these  requirements. 

301.3  Task  Description.  Task  301,  function  certification,  is 
performance  of  a  series  of  qualifying  tests  to  determine  adherence  to  the 
PIDS  requirements.  These  tests  shall  be  designed  to  answer,  as  a  minimum, 
the  following  questions: 

1.  Did  the  (PM,  FD,  FL)  function  detect  the  fault? 

2 .  Did  the  PM  function  indicate  the  proper  operational  status? 

3.  Did  (PM,  FD,  FL)  provide  effective  fault  isolation 
information  for  corrective  maintenance  actions? 

4.  Did  (PM,  FD,  FL)  provide  information  for  further  tests  that 
could  affirm  the  problem? 


22. 


5.  Did  the  PM,  FD  function (s)  provide  information  regarding 
the  impact  of  the  fault  to  the  system? 

6.  What  was  the  latency  time  between  the  occurrence  of  the 
fault  and  the  final  indication  on  the  panel? 

7.  Was  there  any  ambiguity  surrounding  the  fault  or  the 
correction?  (PM,  FD,  FL) 

8.  What  are  the  total  number  of  undetected  faults  in  any  given 
period?  Why  were  they  not  detected? 

9.  Were  there  any  unlocalized  faults?  Why  were  they  not 
localized?  (FL  only) 

10.  What  is  the  latency  time  from  software  glitch  and/or 
hardware  fault  to  automatic  rebooting? 

PM,  FD,  and  FL  design  qualification  is  formal  testing  of  entire 
systems  to  verify  achievement  of  quantitative  PM,  FD,  and  FL  requirements. 
It  cannot  be  combined  with  the  maintainability  demonstration  test,  as  is 
sometimes  tried,  because  the  requirements  of  the  two  tests  differ 
considerably. 

The  maintainability  demonstration  shows  that  maintenance  technicians 
equivalent  to  those  expected  to  maintain  the  system  can  do  so  under  in 
operation  conditions  that  simulate  anticipated  conditions.  During  the 
demonstration,  the  technician  tries  to  restore  the  system  to  operation 
after  each  o~  a  series  of  simulated  faults  is  inserted.  The  test  is  slow 
and  expensive  and  is  usually  based  on  no  more  than  50  fault  simulations. 

The  PM,  FD,  and  FL  demonstration  verifies,  with  high  confidence,  that 
each  of  the  quantitative  PM,  FD,  and  FL  requirements  is  achieved.  More 
than  200  simulations  may  be  required  to  successfully  verify  achievement  of 
some  requirements  with  adequate  coverage  of  all  operational  functions. 

These  simulations  may  be  performed  rather  quickly,  since  actual  repair  need 
not  be  accomplished,  and  the  test  can  be  performed  in  the  laboratory  by 
highly  experienced  engineers.  The  PM,  FD,  and  FL  demonstration  should  be 
performed  before  the  maintainability  demonstration. 

TASK  302  INDEPENDENT  VERIFICATION  &  VALIDATION 

3.2.1  Overview.  Independent  PM,  FD,  and  FL  verification  and  validation 
performed  by  a  scientific  team  not  involved  in  the  design,  development,  and 
tests  ensures  that  the  PM,  FD,  and  FL  designs  meet  the  PIDS  requirements. 
The  independent  IV&V  team  will  ensure  that  the  PM,  FD,  and  FL  subprogram 
will  not  fail  and  will  perform  to  this  intended  capacity. 
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302.2  Purpose.  The  purpose  of  Task  302  is  to  independently  determine 
that  the  PIDS  and  SOW  requirements  have  been  met. 

302.3  Task  Description.  Procedures  shall  be  independently  established, 
maintained  and  implemented,  to  be  performed  by  test  and  analysis,  to 
verify  and  validate  the  ability  of  the  PM,  FD,  and  FL  subsystems  to  meet 
all  of  the  PIDS  and  SOW  requirements.  Functional  testing  of  the  design 
shall  employ  methodologies  of  great  stress  and  strain  to  the  hardware  and 
firmware/software . 

The  PM,  FD,  and  FL  subsystems  shall  be  tested  under  worst-case  actual 
operational  conditions.  The  documentation  produced  by  the  IV&V  team  shall 
include,  but  not  be  limited  to: 

1.  The  test  plan  for  the  tests  that  will  be  conducted, 
including  the  operational  conditions  under  which  the  tests 
will  be  performed. 

2.  The  actual  test  procedures  with  dates,  test  engineer, 
location,  and  all  other  pertinent  information. 

3.  Identification,  description,  listings,  and  source  code  for 
IV&V  test  programs . 

4.  Complete  test  reports,  results,  deficiencies,  problems,  and 
observations. 

The  final  test  of  the  IV&V  test  is  to  be  a  quantitative  test  of  PM, 

FD,  and  FL  capabilities.  This  test  shall  demonstrate  that  all  quantitative 
requirements  of  the  PIDS  have  been  met  at  confidence  levels  that  are 
acceptable  to  the  Government  program  manager. 

TASK  304  PRODUCTION  TEST 

304.1  Overview.  Each  Production  system  will  be  subjected  to  a 
Production  Test  (PT)  to  verify  that  the  system  meets  all  requirements.  The 
PT  will  verify  that  each  system  meets  the  specified  requirements,  and  that 
there  has  been  no  degradation  of  the  processes  used  to  produce  the  EDM 
systems.  Verification  of  continued  achievement  of  PM/FD/FL  requirements 
will  be  made  during  this  test.  The  PT  will  also  be  used  to  collect 
statistical  data  pertaining  to  the  operation  of  the  PM/FD/FL  functions. 

304.2  Purpose .  The  purpose  of  Task  304  is  to  ensure  that  the  PIDS  and 
SOW  PM/FD/FL  requirements  are  met  in  each  delivered  system. 

304.3  Task  Description.  Production  tests  will  be  derived  from  the 
qualification  tests.  The  PT  will  be  used  to  determine  whether  or  not  each 
delivered  system  meets  specified  requirements.  The  PT  will  be  primarily  a 
qualitative  test.  It  will  not  be  sufficiently  extensive  to  permit 
derivation  of  system  parameters  with  a  high  level  of  confidence. 
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SECTION  IV.  CONCLUSIONS/SUMMARY 


The  continuing  increase  in  complexity  of  military  systems  has  imposed 
additional  maintenance  and  logistic  burdens  on  our  operating  forces.  As 
these  organizations  experience  a  reduction  of  both  manning  and  skill 
levels,  the  requirements  for  quick  repair  of  equipment  malfunctions  has 
risen.  Experience  has  indicated  that  when  systems  become  fully 
operational,  a  number  of  problems  are  likely  to  occur  and  that  these 
problems  must  be  dealt  with  in  an  orderly,  precise  and  cost  effective 
manner.  As  a  system  matures,  problems  still  exist  and  all  but  the  simplest 
problems  will  pose  insurmountable  difficulties  to  the  test  and  repair 
technician (s) .  Also,  because  major  turnovers  in  experienced  personnel  is  a 
fact  that  cannot  be  dismissed,  automated  performance  monitoring,  fault 
detection  and  fault  localization  for  sustaining  day-to-day  support  of  a 
system  must  be  accomplished  by  the  user  organization,  namely  our  operating 
fleet.  The  report  presented  herein,  therefore,  presents  a  method  of 
developing  performance  and  maintenance  aid  design  techniques  that  enables 
the  system  to  localize  faults  to  a  manageable  fault  area.  The  technique 
shown  provides  a  record  of  the  design  elements  requisite  to  best  design 
practices  and  provides  a  systematic  approach  to  the  PM/FD/FL  process  not 
previously  provided  in  contract  or  SOW  requirements.  Incorporation  of  this 
document  and/or  portions  thereof  into  system  SOW  documentation  will  allow 
relevant  subject  areas  to  be  addressed  and  judgments  of  conformity  to 
requirements  can  be  more  readily  made  by  the  reviewing  agency. 

The  specification  as  described  herein  has  been  successfully  applied  to 
a  Navy  sponsored  program.  Elements,  as  developed,  were  collected  and 
combined  resulting  in  the  subject  document  for  the  purpose  of  future 
application  in  programs  requiring  PM/FD/FL  design/development. 
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APPENDIX  A 


PM,  FD,  AND  FL  PARAMETERS  AND  THEIR  OPTIMIZATION 


INTRODUCTION 

Performance  Monitoring  (PM) ,  Fault  Detection  (FD) ,  and  Fault 
Localization  (FL)  are  often  independent  functions  within  a  system  or 
equipment,  each  with  a  level  of  sophistication  commensurate  with  the 
needs  of  the  application.  PM,  FD,  and  FL  will  be  treated  as 
separate  functions  in  this  discussion.  The  term  "system"  will  be 
used  throughout  the  discussion. 

It  is  extremely  important  that  PM,  FD,  and  FL  design  be 
performed  simultaneously  with  the  design  of  the  rest  of  the  system. 

Add-on  PM,  FD,  and  FL  functions  will  not  perform  in  an  optimal 

manner,  and  they  will  not  be  as  cost  effective  as  when  designed  as  integral 

functions  of  the  system. 

Some  quantitative  parameters  may  need  to  be  very  high  such  as  an  0.99 
probability  of  success.  In  such  cases,  the  quantitative  requirement  may 
become  the  value  that  can  be  demonstrated  at  a  sufficiently  high  confidence 
level.  While  on  this  subject,  the  reader  is  cautioned  that  the  MIL-STD-471A 
procedure  for  demonstrating  testability  attributes  does  not  always  result 
in  satisfactory  confidence  levels. 

PERFORMANCE  MONITORING 

PM  is  the  function  that  determines  the  integrity  of  a  system.  It 
identifies  unacceptable  system  degradation  resulting  from  combinations  of 
lower  level  deviations  from  tolerances  that  may  be  acceptable  at  the  lower 
levels,  and  it  may  be  designed  to  identify  trends  and  tendencies  to  an 
eventual  failure.  The  PM  function  accomplishes  this  through  injection  of 
known  and  quantified  inputs  into  the  system,  observation  of  responses  for 
expected  performance,  and  reporting  of  deviations  from  expected 
performance.  The  injection  of  test  signals  should  be  planned  so  that  these 
signals  do  not  interfere  with  the  normal  operation  of  the  equipment  or 
system.  The  PM  function  may  also  detect  faults,  but  unless  the  application 
allows  the  combination  of  PM/FD/FL  functions,  fault  detection  by  the  PM 
function  is  synergistic. 

PM  requirements  that  are  to  be  specified  and  optimized  for  the  specific 
application  are  the  probability  of: 

o  A  functional  failure; 

o  Not  diagnosing  a  performance  failure; 

o  Incorrectly  diagnosing  a  performance  fault. 
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FAULT  DETECTION 


FD  is  the  function  that  detects  and  reports  faults.  It  is  usually 
based  on  the  monitoring  of  normally  occurring  outputs  at  selected  test 
points  for  expected  responses.  It  does  not  use  test  signal  injection.  FD 
may,  in  some  cases,  be  combined  with  PM,  but  FD  cannot  be  used  to  perform 
PM.  The  absence  of  faults  is  not  prima  facie  evidence  that  the  system  is 
working  correctly.  FD  should  be  designed  to  detect  all  measurable  faults, 
including  those  that  do 

not  cause  immediate  equipment  or  system  failure.  The  failure  definitions  in 
the  eguipment/system  specif ication  will  define  the  faults  that  cause 
immediate  system  failure,  and  those  that  are  latent  failures.  The  Fault 
Detection  Function  usually  uses  the  "white-  box"  methodology. 

Quantitative  requirements  of  FD  that  must  be  optimized  for  the 
application  include  the  probability  of: 

o  Not  diagnosing  an  existing  fault; 

o  Diagnosing  a  fault  that  does  not  exist; 

o  Incorrectly  diagnosing  the  location  of  a  fault. 

FAULT  LOCALIZATION 

FL  is  the  function  that  isolates  faults  found  by  the  PM  and  FD 
functions  down  to  a  group  of  modules  that  is  small  enough  for  effective 
maintenance.  It  usually  employs  more  comprehensive  tests  than  either  PM  or 
FL.  FL  tests  are  often  invasive  and  require  that  the  equipment  or  system  be 
taken  off-line  for  fault  localization  and  repair.  In  high  reliability 
systems,  the  effect  of  off-line  localization  and  repair  is  minimized  by  the 
use  of  redundant  (parallel)  equipment. 

Quantitative  requirements  of  FL  that  must  be  optimized  for  the 
application  include  the  probability  of: 

o  Not  localizing  the  fault; 

o  Localizing  fault  to  an  incorrect  fault  isolation  group; 
o  Localizing  fault  to  within  the  correct  fault  group. 


DESIGN  OPTIMIZATION 

Basic  design  and  analysis  efforts  of  PM/FD/FL  may  be  based  on  tailored 
application  of  tasks  202  and  203  of  MIL-STD-2165 .  In  addition  to  these 
tasks,  optimization  of  PM/FD/FL  will  require  performance  of  the  additional 
tasks  described  in  this  report. 

The  PM/FD/FL  models  and  predictions  will  be  used  to  optimize  the 
locations  of  test  points  and  plan  for  the  demonstration  and  acceptance 
tests.  They  will  be  derived  from  the  system  reliability  models  and 
predictions,  supplemented  by  information  obtained  from  the  system  Failure 
Modes  and  Effects  Analysis. 
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BACKGROUND 


APPENDIX  B 
GLOSSARY  OF  TERMS 

Those  effects  present  in  physical  apparatus 
or  surrounding  environment  which  limit 
the  measurement  or  observation  of  low  level 
signals  or  phenomenon;  commonly  referred  to 
as  noise. 


BLACK-BOX  METHODOLOGY 


ENHANCED  PM/FD 

FAILURE 

FAULT  DETECTION 

FAULT  IMPACT 

FAULT  ISOLATION  GROUP 

FAULT  LOCALIZATION 

GLITCH 

IMMINENT  FAILURE 

LATENCY  TIME 

LOWEST  REPLACEABLE  UNIT 


The  methodology  which  treats  portions  of 
electronic  units  as  testable  entities.  Known 
and  quantified  inputs  are  injected  into 
entities  being  tested  and  responses  of  the 
entities  to  those  inputs  are  observed  for 
purposes  of  determining  the  integrity  of  the 
entity  under  test. 

Additional  software  and/or  hardware 
incorporated  to  improve  the  probability  of 
detecting  faults. 

A  malfunction  or  combination  of  malfunctions 
that  causes  performance  degradation  below 
acceptable  levels. 

The  PM/FD/FL  function  that  detects  and 
indicates  faults  and  malfunctions. 

A  measurement  of  the  effect  of  a  fault  on 
performance. 

That  module  or  group  of  modules  to  which  a 
fault  is  isolated. 

The  PM/FD/FL  function  which  further  isolates 
faults  found  by  the  performance  monitoring 
and/or  fault  detection  function. 

A  transient  event  that  results  in  the 
improper  operation  of  any  function,  mode  or 
submode  and  that  cannot  be  related  to  a 
specific  hardware  fault. 

Conditions  that  are  likely  to  cause 
functional  failures  if  a  maintenance  action 
is  not  performed. 

The  length  of  time  required  to  detect  and 
identify  a  fault. 

A  unit  which  is  designated  by  the  maintenance 
plan  to  be  removed  upon  failure  from  a  larger 
entity  in  the  latter's  operational 
environment. 
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APPENDIX  B  (CONT'D) 


MONITORED  FAULT 

NON-INVASIVE 

PERFORMANCE  MONITORING 

STRUCTURED  METHODOLOGY 

SYSTEM  INTEGRITY 

WHITE-BOX  METHODOLOGY 


Any  fault  which  will  cause  measurable 
degradation  in  performance  of  any  function 
within  the  system. 

No  adverse  effect  and/or  non-allowable 
degradation  on  system  performance. 


The  PM/FD/FL  function  that  performs  a  macro 
view  of  a  system,  or  part  of  a  system,  in 
order  to  determine  the  health  of  the  portion 
under  test.  Usually  employs  some  sort  of  end 
to  end  testing  with  a  typical  signal 
injection. 

A  method  of  design  which  divides  functions 
into  separate  sub-  functions  that  may  be 
designed,  tested,  and  measured  separately. 

That  state  of  readiness  where  all  system 
functions  meet  all  requirements. 

The  design  methodology  which  states  that  by 
dividing  a  system  into  separate  blocks,  the 
individual  blocks  will  have  internal  test 
points  that  will  adequately  provide  a 
statement  as  to  the  blocks  integrity. 
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APPENDIX  C 


STATEMENT  OF  WORK  SAMPLES 


C-l 


STATEMENT  OF  WORK  FOR  PM/FD/FL 
SAMPLE  ONE 


PERFORMANCE  MONITORING,  FAULT  DETECTION,  FAULT  LOCALIZATION  REQUIREMENTS 


Performance  Monitoring,  Fault  Detection,  Fault  Localization  Program 


The  contractor  shall  develop  and  implement  a  PM/FD/FL  program  in 
accordance  with  the  Statement  of  Work  (SOW) ,  the  Prime  Item  Development 
Specification,  and  NUSC  Technical  Report  8315A  as  tailored  by  this 
Statement  of  Work. 


1 . 1  Elements  of  the  Program 

As  a  minimum,  the  elements  of  the  PM/FD/FL  program  shall  be: 

a.  Task  101  Program  Plan  (PM,  FD,  and  FL) 

b.  Task  102  Monitor/Control  of  Subcontractors/Vendors  (PM,  FD,  FL) 

c.  Task  103  Program  Review  (PM,  FD,  FL) 

d.  Task  104  Configuration  Management  (PM,  FD,  FL) 

e.  Task  201  !. baling  (PM,  FD,  FL) 

f.  Task  202  allocations  (PM,  FD,  FL) 

g.  Task  203  Predictions  (PM,  FD,  FL) 

h.  Task  204  Fault  Tree  Analysis 

i.  Task  205  Fault  Identification  Analysis 

j  .  Task  206  Fault  Impact  Analysis  (PM  and  FD) 

k.  Task  207  Transient  Smoothing  Analysis 

l.  Task  301  Function  Certification  (PM,  FD,  FL) 

m.  Task  302  Independent  Verification  &  Validation  (PM,  FD,  FL) 

n.  Task  304  Factory  Acceptance  Test 

The  contractor  shall  tailor  all  tasks  so  that  there  is  no  duplication  of 
work  performed  under  tasks  such  as  Failure  Modes  and  Effects  Analysis, 
Logistics  Support  Analysis,  and  Maintainability  Prediction. 
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STATEMENT  OF  WORK  FOR  PM/FD/FL  (CONT'D) 
SAMPLE  ONE 


2 . 0  Applicable  Documents 

The  following  documents  are  applicable  to  PM/FD/FL  to  the  extent 
specified  herein.  In  case  of  conflict  between  the  Contract  Schedule,  the 
SOW,  and  the  applicable  documents,  the  order  of  precedence  in  the  SOW  shall 
apply. 

(Note:  Here  the  documents  as  required  by  the  particular  program  would 
inserted) . 
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STATEMENT  OF  WORK  FOR  PM/FD/FL 
SAMPLE  TWO 


PERFORMANCE  MONITORING,  FAULT  DETECTION,  FAULT  LOCALIZATION  REQUIREMENTS 


1.0  Performance  Monitoring,  Fault  Detection,  Fault  Localization  Program. 

The  contractor  shall  develop  and  implement  a  PM/FD/FL  program  in 
accordance  with  the  Statement  of  Work  (SOW) ,  the  Prime  Item  Development 
Specification,  and  Section  3.0  of  this  Appendix  to  the  SOW.  The  program 
shall  be  tailored  to  the  needs  of  the  project  from  the  requirements  of  NUSC 
Report  Number  8315A.  The  contractor  shall  also  tailor  each  task  so  that 
there  is  no  duplication  of  work  performed  under  tasks  such  as  Failure  Modes 
and  Effects  Analysis,  Logistics  Support  Analysis,  and  Maintainability 
Prediction. 

2 . 0  Applicable  Documents 

The  following  documents  are  applicable  to  PM/FD/FL  to  the  extent 
specified  herein.  In  case  of  conflict  between  the  Contract  Schedule,  the 
SOW,  and  the  applicable  documents,  the  order  of  precedence  in  the  SOW  shall 
apply. 

3 . 0  Requirements 


The  contractor  shall  perform  the  tasks  of  NUSC  Report  8315A  as 
modified  and  tailored  herein. 

4.0  Software/ Firmware  and  Hardware  Development,  Verification,  Validation 

The  PM/FD/FL  subsystem  shall  be  developed,  verified,  and  validated 
in  accordance  with  the  following  sections  of  this  SOW: 

a.  Software  and  Firmware:  Software  Development  requirements 

b.  Hardware:  Reliability,  Maintainability,  and  Quality  Assurance 
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APPENDIX  D 


SAMPLE  CDRL  ITEMS 


CONTRACT  DATA  REQUIREMENTS  list 
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APPENDIX  E 


I 


APPLICABLE  DATA  ITEM  DESCRIPTIONS 


form  Approved 

DATA  ITEM  DESCRIPTION REC'B  SEP  IS  1985 


1.  IDENTIFICATION  NUMBER 


HARDWARE  DIAGNOSTIC  TEST  SYSTEM  DEVELOPMENT  PLAN 


DI-ATTS- 80005 


3  OESCR.PTiON/PURPOSE 

3.1  The  Hardware  Diagnostic  Test  System  (HDTS)  Development  Plan  describes  the  con¬ 
tractor’s  plan  for  developing  and  integrating  a  hardware  fault  diagnostic  and  test 
capability  for  system/subsystem/equipment.  It  provides  a  controlled  statement  of  the 
contractor's  plan  for  producing  and  developing  the  diagnostic  software  and  hard¬ 
ware  diagnostic  test  devices  which  satisfy  the  functional,  performance,  and 


6«.  OTIC  REQUIRED 


.  6b.  GiDEP  REQUIRED 


«  APPROVAL  DATE  5  OFFICE  OF  PRIMARY  RESPONSIBILITY  10PR)  wr..v  —  — - - 

TrYMMOOj  G/T213 

850610 _ _ _ _ _ _ _ _ 

7.  APPLICATION /INTERRELATIONSHIP 

7.1  The  Hardware  Diagnostic  Test  System  Development  Plan  provides  the  contractor 
with  the  means  to  coordinate,  control,  and  monitor  progress  of  the  development  ef¬ 
fort.  It  provides  the  Government  with  knowledge  of  the  schedule,.  organization  and 
resource  allocation  planned  by  the  contractor.  It  is  a  basic  tool  with  which  the 
Government  can  monitor  the  contract  work  effort. 

7.2  This  data  item  description  (DID)  satisfies  the  requirements  of  paragraph  5.1, 

DOD-STD-1701  (NS)  _ _ _ _ 

9b.  AMSC  NUMBER 


8  APPROVAL  LIMITATION 


10  PREPARATION  INSTRUCTIONS 


9*.  APPLICABLE  FORMS 


G3611 


10.1  Source  document.  This  applicable  issue  of  the  document  cited  herein,  including 
its  approval  date  and  dates  of  any  applicable  amendments  and  revisions,  shall  be  as 
reflected  in  the  contract. 

10.2  The  HDTS  development  plan  shall  consist  of  ten  sections  with  appropriate 
subsections.  The  format  shall  be  as  follows. 


Section  I 


-  Introduction 


Section  II 


-  Organization  and  Responsibility 


Section  III  -  Management  and  Technical  Controls 

Section  IV  -  Resources 

4.1  Personnel 

4.2  Training 

4.3  Data  Processing  Equipment 


Section  V 


-  Software  Development  Schedule 


DO  form  1664,  FEB  85 


•Vtwouf  ftfioon  «f  OOSOtfft 
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3.  DESCRIPTION/PURPOSE  (Cont'd) 

operational  requirements  of  the  system/subsystem/equipment .  It  is  used  to  approve 
the  contractor's  approach  for  a  Hardware  Diagnostic  Test  System  (HDTS),  and  to  mon¬ 
itor  and  evaluate  the  contractor's  progress  while  developing  the  HDTS. 


10.  PREPARATION  INSTRUCTIONS  (Cont'd) 

Section  VI  -  Monitoring  and  Reporting 

Section  VII  -  Documentation 

Section  VIII  -  Development  Approach 

8.1  Engineering  Practices 

8.2  Operating  Practices 

Section  IX  -  Development  and  Test  Tools 

Section  X  -  Security  Controls  and  Requirements 

10.3  The  content  of  each  section  shall  be  as  follows. 


10.3.1  Section  I.  Introduction.  This  section  shall  describe  the  scope,  purpose, 
application  and  authority  of  the  development  effort.  This  should  include  a  brief 
overview  of  the  management  philosophy  and  methodology  that  will  be  used  on  the 
project. 


10.3.2  Section  II.  Organization  and  Responsibility.  This  section  shall  describe 
the  organization,  responsibilities  and  structure  of  the  groups  that  will  be  design¬ 
ing,  producing  and  testing  all  segments  of  the  software  system.  It  shall  also 
identify  the  name  and  management  position  of  each  supervisor. 

10.3.3  Section  III.  Management  and  Technical  Controls.  This  sections  shall 
describe  the  management  and  technical  controls  that  will  be  used  during  development, 
including  controls  for  insuring  that  all  performance  and  design  requirements  have 
been  identified  and  implemented. 

10. 3.  A  Section  IV.  Resources. 

10.3. A.1  Personnel.  This  section  shall  identify  the  level  of  manpower  allocated  to 
each  task  shown  in  the  development  schedule,  Including  numbers,  duration  of  as¬ 
signment,  and  required  skills.  This  includes  administrative  and  logistic  support 
personnel.  If  known,  personnel  assigned  to  software  development  tasks  shall  be 
Listed  by  name.  This  section  shall  also  identify  security  clearance  requirements  and 
ilans  for  obtaining  the  necessary  security  clearances  for  personnel  working  on  the 
;oftware  system  (if  applicable). 

0.3. A. 2  Training.  This  section  shall  identify  training  required  for  people  working 
m  the  project  and  dates  by  which  the  training  must  be  completed. 
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10.  PREPARATION  INSTRUCTIONS  (Cont'd) 

10. 3. A. 3  Data  Processing  Equipment.  This  section  shall  identify  requirements  for 
the  use  of  data  processing  equipment  to  support  the  development  of  computer  programs 
and  their  subsequent  testing.  It  shall  also  describe  the  plan  for  assuring  that  the 
necessary  hardware  is  available  at  the  appropriate  times. 

10.3.5  Section  V.  Software  Development  Schedule.  This  section  shall  present  a 
graphic  and  narrative  description  of  the  scheduled  events  and  milestones  of  the  soft¬ 
ware  development  effort.  The  schedule  will  be  updated  to  reflect  additional  detail 
as  the  project  moves  through  successive  phases  of  the  development  cycle.  By  Pre¬ 
liminary  Design  Review,  this  section  shall  include  a  development  schedule  for  each 
computer  program  and  data  base.  The  graphic  description  shall  be  a  chart  identifying 
schedules  for  the  following: 


a . 

All  deliverables; 

b. 

Preparation  of  management 

and  test  plans; 

c . 

All  levels  of  testing; 

d. 

Reviews,  including  major 

reviews 

and  other  internal  milestones; 

e . 

Transition  to  life-cycle 

support 

activity. 

The  chart  should  illustrate  a  relationship  with  hardware  schedules.  Critical 
paths  shall  also  be  identified. 

IU.3.6  Section  VI.  Monitoring  and  Reporting.  This  section  shall  describe  the  pro¬ 
cedure  for  monitoring  and  reporting  the  status  of  program  development.  It  shall  also 
describe  the  manner  in  which  problems  and  recommended  solutions  to  problems  will  be 
reported . 

10.3.7  Section  VII.  Documentation.  This  section  shall  describe  the  approach  for 
developing  computer  program  documentation  and  will  identify  the  documentation  that 
will  be  produced.  This  shall  include  the  plan  for  developing  test-planning  documen¬ 
tation,  the  Software  Requirements  Specification,  the  System/Subsystem  Specification, 
the  Program  Specification,  Software  Manuals  and  any  other  documentation. 

10.3.8  Section  VIII,  Development  Approach. 

10.3.8.1  Engineering  Practices.  This  section  shall  describe  the  engineering  prac¬ 
tices  that  will  be  applied  to  the  development  of  software.  These  practices  Include 
standards,  conventions,  procedures,  rules  for  programming,  design  and  other  disci¬ 
plines  affecting  development.  At  a  minimum,  procedures  for  implementing  the  follow¬ 
ing  practices  shall  be  described: 

a.  Programming  and  data  base  standards; 

b.  Top-down  design  methodology; 

c.  Design  walk-throughs. 
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PREPARATION  INSTRUCTIONS  (Cont'd) 

10.3.8.2  Operating  Practices.  This  section  shall  describe  the  operating  practices 
that  will  be  applied  to  the  development  of  software.  These  include  the  following; 

a.  Use  of  Unit  Development  Folders; 

I 

b.  Techniques  for  ensuring  that  all  performance  and  design  requirements  have 
been  Implemented; 

c.  Means  of  ensuring  modularity,  ease  of  modification,  and  capacity  for  com¬ 
puter  program  growth; 

d.  Methods  and  procedures  for  collecting,  analyzing,  monitoring  and  reporting 
on  the  timing  of  time-critical  computer  programs; 

e.  Means  for  ensuring  that  the  software/data  processors/peripheral  equipment 
interfaces  are  adequate; 

f.  Criteria  for  determining  when  a  development  unit  should  be  entered  into 
configuration  control; 

g.  Means  of  controlling  master  copies  of  computer  programs,  data  bases  and 
associated  documentation  during  development  (including  their  relationship  to  the  Con¬ 
figuration  Management  Plan); 

h.  Rules  for  Interface  definition. 

10.3.9  Section  IX.  Development  and  Test  Tools.  This  section  shall  identify  the 
special  tools  and  techniques  that  will  be  used  during  development  and  testing  of  the 
computer  programs.  Some  examples  are  as  follows: 

a.  Special  simulation; 

b.  Data  reduction; 

c.  Code  optimizers; 

d.  Code  auditors; 

e.  Special  utility  programs; 

f.  Software  security  test  tools. 

10.3.10  Section  X.  Security  Control  and  Requirements.  This  section  shall  identify 
security  controls  that  will  be  used  during  software  development  (e.g.,  physical 
security,  document  access  controls,  computer  access  controls,  etc.).  It  6hall  also 
describe  the  method  of  implementing  and  maintaining  the  security  controls.  It  shall 
also  Identify  and  unique  security  problems  and  installation  security  requirements. 
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1.  IDENTIFICATION  NOUt.  | 

DATA  ITEM  DESCRIPTION 

AGENCY 

NUMBER 

1.  TITLE 

DESIGN  REVIEW  DATA  PACKAGE 

MSA 

DI-E-5423 

».  OUC«'PTtOH/PUBPOU 

3.1  The  data  packages  are  required  by  the  Government  to 

4.  APPROVAL  DATE 

1977  May  02 

permit  adequate  preparation  for  each  design  review  prior 
to  the  review  meeting. 

1 

a.  OFFICE  OF  PRIMARY 

NSA-R41 

4.  APPROVAL  LIMITATION 

i 

1.  *r  PL1C  ATION/1NT CHHCL  ATIOWSM1P 

7.1  To  be  used  on  contracts  which  require  formal  technical 
reviews  and  audits. 

I.  IHrEMHCII  u  c»>4  In 

flock  It) 


MIL-STD-1521 


HCIL  NUMStnin 


10.  OMCOANATION  INITNUC  f  ION! 

10.1  Data  packages  shall  be  provided  for  design  review  meetings  to  be  held  on  the 
program  and  submitted  as  Indicated  on  DD  Form  1423.  The  data  packages  shall  be 
designed  to  provide  adequate  preparation  information  for  design  reviews  organized 
in  accordance  with  MIL-STD-1521  and  Appendices  B,  C,  D,  and  G.  The  detail  contents 
of  each  package  shall  include,  but  not  be  limited  to,  the  material  required  for  the 
subject  design  review,  an  agenda,  and  a  status  of  pertinent  (.if  any)  action  item9 
from  previous  design  reviews  or  other  meetings. 
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DATA  ITEM  DESCRIPTION 


I.  TITLE 

Configuration  Management  Plan  (CMP) 


a.  oEteRiPTiON/puppotE 

This  plan  is  prepared  by  the  contractor  to  describe  his 
assignment  of  responsibilities  organizationally  and  the 
procedures  used  in  hie  accomplishipent  of  the  specific  1 
configuration  management  requirement  as  stated  in  the 
contract.  It  is  not  to  be  used  as  a  contractual  require¬ 
ment  In  lieu  of  the  statement  of  work. 


AP PLIC  ATION/INTBRREL  ATIONENIP 

Obtained  as  part  of  the  validation  phase  final  report. 
When  a  validation  phase  is  not  accomplished,  the  CMP 
will  be  a  requirement  of  the  full-scale  development 
contract.  Not  to  be  used  on  follow  on  contracts  where 
the  contractor's  configuration  management  organization 
and  procedures  have  been  satisfactorily  demonstrated 
on  prior  contracts.  This  DID  may  be  modified  and  used 
on  competitive  RFPs  to  acquire  information  for  source 
selection.  When  used  in  this  manner,  only  an  abbreviat 
ed  plan  will  be  acquired.  By  the  aime  token,  when  this 
plan  is  procured  (on  other  than  validation  contracts)  it 
should  be  modified  to  delete  source  selection  require¬ 
ments. 


AOCHCY 

NUMBER 

USAF 

DI-E-3108/ 

C-1I8-1 

4.  APPROVAL  DATS 

26  February  1971 

*.  doc  required 


APPROVAL  LIMITATION 


RereneNCKi  (Mmdmtory  e  Ifd 
In  bleclt  10) 

MIL-STD-483  (IJSAF) 
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10.  PREPARATION  INSTRUCTION* 

The  contractor  shall  describe  in  a  configuration  management  plan,  the 
organizational  responsibilities  and  procedures  used  in  the  implementation  of 
the  configuration  management  requirements  as  stated  in  the  contract.  The 
configuration  management  plan  she  ll  be  prepared  in  accordance  with  the 
criteria  set  forth  in  Appendix  I  of  MIL-STD-483  (USAF). 
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Maintainability  Modeling  Report 

DI-NNTY 

-80823 

[x  oc*c**r«ON/*u*#ost 

3.1  To  describe  and  show  the  development  of  a  matntalnabll Ity  model  for  making 
numerical  maintainability  apportionments  to  various  functions  and  levels  of  hard¬ 
ware  throughout  an  item  (system,  subsystem,  equipment)  and  to  evaluate  the  main¬ 
tainability  of  an  item  based  on  fts  maintainability  design  characteristics. 
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7.1  This  DID  contains  the  content  and  format  requirement  of  the  data  item  generated  by  paragraph . 
201.2  of  Task  201  of  MIL-STD-470B.  This  DID  is  applicable  to  contracts  which  contain  the 
requirments  for  Task  201  "Maintainability  Modeling,"  of  MIL-STD-470B. 


7.2  This  DID  supersedes  DI-R-71 06. 


1  A4MOVAI  UMITATIOW 

fa  4MUC4IU  404  MS 

fa  AM$C  NUMII* 
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10  7K(AaJlaTiON  INSTHUCTtONS 


10.1  Reference  documents.  The  applicable  issue  of  the  documents  cited  herein, 
including  their  approval  dates  and  dates  of  any  applicable  amendments,  notices,  and 
revisions,  shall  be  as  specified  in  the  contract. 

10.2  Content.  The  report  shall  contain  the  Halntainabil ity  model(s)  developed  in 
accordance  with  paragraph  201.2  of  Task  201  "Maintainability  Modeling"  of  MIL-STD- 
470  as  tailored  to  the  particular  needs  of  the  acquisition  program. 


10.3  Format.  The  report  shall  be  prepared  in  accordance  with  ANSI  Z39.18, 
"Scientific  and  Technical  Reports:  Organization,  Preparation,  and  Production. 


11 .  OllTIMUTlOM  STATfMCNT 

DISTRIBUTION  STATEftNT  A.  Approved  for  public  release;  distribution  is  unlimited. 
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DATA  ITEM  DESCRIPTION 


Form  Approvt  <? 
OMB  Ho  0704-01  tS 


Maintainability  Allocations  Report 


2  IDENTIFICATION  NUMBED 


DI-MNTY-80826 


3.  DESCRIPTION /  PURPOSE 


3. 1  To  document  the  quantitative  and  quafitaiive  maintainability  requirements  developed  for  each 
component  item  of  the  approved  hardware  breakdown  structure  derived  to  meet  the  end  item  requirements. 


I  (T  DTre  APPLICABLE  l«b  ClDEP  APPLICABLE 


I  4.  APPROVAL  DATE  5.  OFFICE  Of  PRIMARY  RESPONSIBILITY  (OPR!  El  DTrC  APPLICABLE  *b.  OlDEP  APPUCAB 
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7.  APPLICATION /INTERRELATIONSHIP 

7.1  This  DID  contains  the  content  and  format  requirements  of  the  data  item  generated  by  paragraph  202.2  of 
Task  202  of  MIL-STD-470B.  The  DID  is  applicable  wheneverTask  202  is  called  out  as  pan  of  an  acquisition 
program.  System/subsystem/equipmenl  level  quantitative  maintainability  requirements  must  be  broken 
down  to  appropriate  subsysterrVequipmentAinit/subunit  levels  as  necessary  to  establish  requirements  lor 
designers  and  subcontractors. 


7.2  This  DID  supersedes  DI-R-71 07. 


I  (.  APPROVAL  LIMITATION 


r»t  APPLICABLE  FORMS 


9b  AMSC  NUMBER 

F4711 


10.  PREPARATION  INSTRUCTIONS 

1 0.1  Reference  documents.  The  applicable  issue  of  the  documents  cited  herein,  inducing  their  approval 
dates  and  dates  of  any  applicable  amendments,  notices,  and  revisions,  shal  be  as  specified  in  the  contract. 

10.2  Content.  The  Maintainability  Allocations  Report  shall  include  the  information  required  by  paragraph 

202.2  of  Task  202  of  MtL-STD-470,  as  tailored  for  the  particular  acquisition.  The  report  shall  provide  the 
results  and  describe  the  process  of  allocating  maintainability  requirements  to  each  component  end  item. 

10.3  Format.  The  format  of  the  report  shall  be  in  accordance  with  ANSI  Z39.1 8,  “Scientific  and  Technical 
Reports:  Organization,  Preparation,  and  Production". 
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DISTRIBUTION  STATEMENT  A.  Approved  for  public  release;  distribution  is  unBmited. 
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DATA  ITEM  DESCRIPTION 

Form  Approvtd 

OMB  No.  0704-C1H 

1.  Title 

Maintainability  Predictions  Report 

2.  IDENTIFICATION  NUMBER 

DI-MNTY-80827 

3  DESCRIPTION /PURPOSE 

3.1  To  provida  the  description  and  documentation  of  the  mainlainabflty  prediction  made  by  the  contractor. 
To  make  a  determination  ol  whether  or  not  the  proposed  design  is  consistent  with  maintainabiRty 
requirements. 


4  APPROVAL  DATE 

rmnwoot 
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890530 
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7.  APPLICATION  /INTERRELATIONSHIP 


0 

7.1  This  DID  contains  the  content  and  format  requirements  of  the  data  Kern  generated  by  paragraph  203.2  of 
Task  203  of  MIL-STD-470B.  The  content  of  this  report  shall  be  included  in  the  Maintainability  Predictions 
Report  of  Mll-HDBK-472  when  that  has  been  designated  as  the  basis  for  Task  203  of  MIL-STD-470B. 

7.2  This  DID  supersedes  DI-R-7108. 


t  APPROVAL  LIMITATION 


9*  APPLICABLE  POP  MS 


9b  AMSC  NUMBER 

F4712 


10.  PREPARATION  INSTRUCTIONS 

10.1  Reference  documents.  The  applicable  issue  of  the  documents  cited  herein,  including  their  approval 
dales  and  dates  of  any  applicable  amendments,  notioes,  and  revisions,  shall  be  as  specified  in  the  oontract. 

1 0.2  Content.  The  Maintainability  Predictions  Report  shall  contain  the  following  detail  as  tailored  for  the 
particular  acquisition: 

a  Assumptions  used  in  the  prediction  prooess. 

b.  Identification  of  the  prediction  procedure  used. 

c.  Prediction  results  to  the  appropriate  levels. 

10.3  Format.  The  format  of  the  report  shall  be  in  aocordance  with  ANSI  Z39.18.  ’Scientific  and  Technical 
Reports:  Organization,  Preparation,  and  Production’. 


11  DISTRIBUTION  STATEMENT 


DISTRIBUTION  STATEMENT  A  Approved  for  public  release;  distribution  is  unimited. 
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Scientific  and  Technical  Reports  Summary  DI-MISC-80048 


3.  OESCRIPnON/RURPOSE  '  — 

3.1  Technical  reports  are  acquired  to  provide  the  scientific  and  technical 
community  a  description  of  the  precise  nature  and  results  of  research,  develop¬ 
ment,  test,  and  evaluation  (RDT&E).  accomplished.  Technical  r.eports  may  be  defin¬ 
itive  for  the  subject  presented,  exploratory  in  nature,  or  an  evaluation  of  criti¬ 
cal  subsystem  or  of  technical  problems. 


*  APPROVAL  OATE  5.  OFFICE  OP  PRIMARY  RtSPONSIHUTY  (OPR)  fe.  OTIC  REQUIREO  to.  GIOIP  REQUIRED 
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7.  APMJCAnON/INrtRRElAnoNSMIP 

7.  1  This  Data  Item  Description  contains  the  data  format  and  content-  preparation 
instructions  for  the  data  product  generated  by  the  specific  and  discrete  task 
requirements  for  this  data  included  in  the  contract. 

7.2  This  Data  Item  Description  shall  be  used  in  preparing  all  ongoing  interim  or 
final  Scientific  and  Technical  Reports  Summary.  The  purpose  of  these  report  sum¬ 
maries  is  to  present  management  with  a  concise  description  of  the  scientific  and 
technical  findings  and  accomplishments  during  the  reporting  period. 


1.  AARROVA L  UMIfAHON 

i 
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to.  AMSC  NUMIER 

A3670 

10.1  Contract.  This  Data  Item  Description  is  generated  by  the  contract  which 
contains  a  specific  and  discrete  work  task  to  develop  this  data  product. 

10.2  Format.  The  Scientific  and  Technical  Reports  Summary  shall  be  in  contractor 
format. 

10.3  Contents.  The  level  of  detail  or  the  Scientific  and  Technical 
Reports  Summary  shall  be  adequate  for  non-specialists  in  the  subject  matter. 

When  appropriate,  specific  references  should  be  made  to  more  detailed 
materials.  The  content  of  the  Scientio  and  Technical  Report  Summary  shall 
consist  of  the  following: 

(a)  Task  objectives. 

(b;  Technical  problems. 

(c)  General  methodology  (e.g.,  literature  review,  lab  experiment, 
survey,  etc). 

(d)  Technical  results. 

(e)  Important  findings  and  conclusions. 
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Scientific  and  Technical  Reports  Summary  (Cont'd) 

Block  7  APPLICATION/INSTRUCTIONS  (Cont'd) 

7.2  (Cont'd)  The  types  of  scientific  and  technical  report  summaries  and  their 
frequencies  are  specified  in  the  DD  Form  1423 

7.3  This  Data  Item  Description  shall,  be  applicable  in  contracts/  when  DI-S-^OST 
is  used. 


Block  10  PREPARATION  INSTRUCTIONS  (Cont'd) 

10.3  (Cont'd) 

(f)  Implications  for  futher  research 

(g)  Significant  hardware  development 

(h)  Special  comments 

10.4  Cover  Page  -  The  heading  or  cover  page  of  each  report  summary  shall 
contain  the  following  information: 

(a)  Procuring  Activity  Designated  Order  Number 

(b)  Name  or  Contractor 
(o)  Contract  Number 

(d)  Effective  Date  of  Contract 

(e)  Expiration  Date  of  Contract 

(f)  Reporting  Period 

(g)  Principal  Investigator  and  Phone  No. 

(h)  Project  Scientist  or  Engineer  and  Phone  No 

(i)  Short  Title  of  Work 

10.4. 1  Additionally,  each  report  produced  will  have  prominently  displayed  on  the 
oover  page,  a  notioe  of  disclaimer  worded  as  follows: 

The  views  and  oonolusions  oontained  in  this  dooument  are  those  of  the 
authors  and  should  not  be  interpreted  as  neoessarily  representing  the  offical 
polioies,  either  expressed  or  implied,  of  the  Government. 

10.11.2  Scientifio  and  Technical  Reports  which  are  sponsored  by  other  than  the  pro¬ 
curing  aotivity  shall  have  the  following  on  the  front  oover: 
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Scientific  and  Technical  Reports  Summary  (Cont'd) 

Block  10  PREPARATION  INSTRUCTIONS  .  (Cont'd) 

Sponsored  by 

(Sponsor's  Identification) 


(Sponsor's  Designated)  Order  No. 
Monitored  by  _ 


/ 

Under  Contract#  _ 


10.5  Reports  shall  be  reproduced  only  by  prooesses  which  provide  black  on  white 
copy  sufficiently  clear  and  sharp  for  further  reproduction  when  required.  Ditto, 
hectograph,  color,  and  other  reproduction  processes  not  reproducible  photograph¬ 
ically  or  xerographlcally  are  not  acceptable. 
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This  daca  item  is  used  Co  der.cribe  a  contractor's  test 
procedure  and  how  he  intends  to  determine  compliance  vith 
specification  requirements. 


74  Oct  23 
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I.  BOC  NKflUl 


«.  AV.  ATI©* 


1.  APPUCATlOH/IMTIAMCAflOMiKlP 


Application  vlLl  be  as  specified  by  the  contract  daca 
requirements  list.  This  item  toay  be  used  whenever  tests 
are  required. 


I.  Rif  (RINCII  (M^Riienr  *i  (MW  M 
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wen  MuuiiRia 
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••  PRCARATIOM  INITRUCTIONI  ! 


iO-i  The  test  procedures  r.hall  be  typed  in  contractor  or  commercial  format  on 
8"xlOV  sheets. 

10.2  The  tesc  procedures  shall  cover  in  derail  the  plan  and  procedures  for  acccoipli*:. 
ment  of  the  rests  specified  in  the  contract  schedule  and  specif icatinns  referenced 
therein  or  in  Block  16  of  the  DD  Form  1623*  Contract  Data  Requirements  List,  detu 
item  requiring  these  procedures  and  shall  specifically  cover  or  contain  the  fol Irving 
as  applicable: 


a.  Title 

b.  Index 

c.  Identification  of  item  being  tested  (serial  number) 

d.  Identification  number  of  teat  procedure 

e.  Hardware  configuration 

f.  Test  p.  (requisites 

g.  Report  form 

h.  Date,  time  and  duration  of  test 

I.  Proposed  test(s) 

J.  PreoperatlooaL  checklist 
k.  The  purpose  of  che  test (s) 


(JDI-T-23732B 


PROCLDLRES ,  TEST  (Con.  Anued) 

l.  Description  of  test 

m.  The  specification  paragraph(s)  to  which  the  test(s)  will  prove 
compliance. 

n.  Detailed  step-by-step  procedure  (may  be  referenced  to  test  number 
and  cest  title  in  Government  documents) 

o.  Test  schedule  (operating  profile,  setpoints,  stabilization  time, 
data  points) 

p.  The  cest  equipment  utilized. 

q.  Approvals,  authorities  and  responsibilities 

r.  Sketches  or  photographs  or  test  set-up 

s.  Facilities  required  for  test 

c.  Test  equipment  requir eaeiits  (major  and  special) 

u.  Methods  of  measurement (s) 

v.  Logistics  equipment  requirements  (spare  test  hardware) 

v.  Method  of  control  of  sub-contractor's  efforts  and  their  procedures. 

x.  Applied  instrumentation  and  data  recording  equipment 

y.  Data  sheets  (when  required  by  a  specif ication)  for  which  the  results 
ate  able  to  be  correlated  to  the  item  tested. 

z.  Types  of  data  to  be  recorded  (parameters,  ranges,  accuracies,  type 
readout,  and  quantities) 

aa.  Results  (-.tonparieon  of  test  data  to  acceptance  standard) 
bb.  Accept." reject  criteria  for  test  acceptance, 
cc.  Personn-il  required 
dd.  Special  resource  requirements 

ee.  References  to  specs,  standards,  tech  manuals,  other  test  procedures 

and  reports,  change  orders,  notices,  and  other  references  not  specific 
tc  the  t  st  but  Included  for  information  only. 

In  addition  r.c  the  requirements  of  paragraph  10.2,  the  production  test 
edures  s... —  covur  cleanin’/refurbishing  of  test  equipment  and,  if  applicable, 
rrclar lonship  for  and  during  availability  test(s). 
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Action  System,  Reports 

2.  iMNTVlCATtOM  H 

D1-MK1 

IMIU 

ry-60824 

J.  0f]C*IRT1OM>  PU«POH  ~ 

3.1  This  data  Is  used  to  aid  maintainability  design.  Identify  corrective  action 
tasks  and  to  evaluate  test  results.  The  reports  generated  shall  consist  of  tabula¬ 
tions  and  analyses  of  all  maintenance  actions  occurring  through  the  reporting 
period  as  well  as  remedial  actions  proposed  by  the  contractor  to  eliminate  main¬ 
tainability  deficiencies  (and  fault  detectlon/lsolatlon  deficiencies). 

4.  APPROVAL  OA  fC 
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7.1  This  DID  contains  the  content  and  format  requirements  of  the  data  item  generated  by  paragraph 
104.2  of  Task  104  of  MIL-STD-470B.  This  DID  is  appScable  when  Task  104,  "Data  Collection,  Analysis 
and  Corrective  Action  system"  of  MIL-STD-470B  is  called  out  as  part  of  the  acquisition  program. 

This  DID  should  be  prepared  in  conjunction  with  the  "Maintainabifity  Demonstration  -Reports"  called 
out  in  MIL-STD-471  A. 


7.2  This  DID  supersedes  Di-R-71 05. 
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10.1  Reference  documents.  The  applicable  Issue  of  the  documents  cited  herein, 
Including  their  approval  dates  and  dates  of  any  applicable  amendments,  notices,  and 
revisions,  shall  be  as  specified  In  the  contract. 

10.2  Content.  The  report  content  shall  describe  the  results  of  the  "Data  Collec¬ 
tion,  Analysis  and  Corrective  Action  System". 


a  The  report  shall  include  subcontractor,  vendor  data,  as  applicable. 

b.  Data  coTlected,  analyzed  and  documented  should  be  representative  of  the 
Information  elements  contained  below: 


(1)  A  maintenance  event  identification  number. 

(2)  Maintenance  task  Identification,  keyed  to  each  maintenance  event 
(detection.  Isolation,  removal,  checkout,  etc.) 

(3)  Date  on  which  the  maintenance  event  took  place. 

(4)  Identification  of  the  location  where  the  maintenance  event  took 

place. 


(Continued  on  Page  2) 
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Block  10,  Preparation  Instructions  (Continued) 

(5)  Identification  of  system,  subsystem,  assembly,  printed  circuit  card 
on  which  maintenance  was  performed. 

(6)  Maintenance  time  necessary  for  corrective  actions  (or  maintenance 
manhours,  where  appropriate). 

(7)  Deficiencies  found/ corrective  actions  taken. 

(8)  Diagnostic  effectiveness  data  (percent  of  fault  detectable,  iso¬ 
late  ble,  false  alarm  rates,  etc.). 

(9)  Logistic  Support  Analysis  (LSA)  applicable  data. 

10.3  Format .  The  report  shall  be  prepared  In  accordance  with  ANSI  239.18, 
"Scientific  and.  Technical  Reports:  Organization,  Preparation,  and  Production". 
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Testability  Program  Plan 


[».  O  CSC  MINT  IOM/  PURPOSE 


3.1  This  plan  Identifies  the  performing  activity  approach 
for  Implementing  a  Testability  Program  in  accordance 
with  MIL-STD-2165. 


DI-T-7198 


4.  SPPSDVAL  DATf 

29  January  1985 

S-  omrtCi  DP  PKiMisv 
MIPONMSIUrT 

NAVY-EC 


Is.  DOC  SCOUIRCO 


IS.  APPRSVAL  UMTATISN 


T.  APPV.IC  ATIOM/INTSNMCL  ATIOMSMlN 

7.1  These  data  are  to  be  used  to  define  a  Testability 
Program  Plan. 

7.2  This  DIO  may  be  used  for  all  electronic  system  and 
equipment  development  programs. 

7.3  This  DID  satisfies  the  data  requirements  of  Task  101 
of  MIL- STD-21  65. 
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’O.  *A<PAPAT>OM  INSTRUCTIONS 


AMSC  NO.  N3424 


10.1  The  applicable  Issue  of  the  documents  cited  herein,  including  their  approval 
dates  and  applicable  amendments  and  revisions,  shall  be  as  reflected  In  the 
contract. 

10.2  Contractor's  format  Is  acceptable. 

10.3  A  Testability  Program  Plan  shall  be  prepared  In  accordance  with  NIL-STD-2165, 
Task  101  and  include  the  following  elements,  with  the  range  end  depth  of 
Information  for  each  element  tailored  to  the  acquisition  phase: 

10.3.1  A  description  of  the  work  to  be  accomplished  for  each  testability  task 
Included  in  the  contractual  requirements. 

10.3.2  The  time  phasing  of  each  task  and  its  relationship  to  other  tasks, 
particularly  maintainability  tasks. 

10.3.3  Identification  of  a  single  organizational  element  within  the  performing 
activity  which  has  overall  responsibility  and  authority  for  Implementa¬ 
tion  of  the  testability  program. 

10.3.4  Identification  of  data  Interfaces  between  the  organizational  element 
responsible  for  testability  and  other  related  elements. 
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Testability  Program  Plan 

10.  Preparation  Instructions  (Cont'd) 

10.3.5  Identification  of  the  method  by  which  testability  requirements  will  be 
integrated  with  other  design  requirements  and  disseminated  to  design 
personnel  and  subcontractors. 

10.3.6  Identification  of  testability  design  guides  and  testability  analysis 
procedures  to  be  used. 

10.3.7  Description  of  procedures  for  scheduling,  conducting  and  documenting 
testability  design  reviews. 

10.3.8  Identification  of  testability  submissions  and  their  review,  verification 
and  utilization. 

10.3.9  Description  of  procedures  for  identifying  testability-related  problems 
and  assuring  corrective  action. 

10.3.10  Description  of  procedures  and  controls  for  assuring  that  each  subcontractor* 
testability  practices  are  consistent  with  overall  system  or  equipment 
requirements. 
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2.  IDENTIFICATION  NOISI.  | 
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AGENCY 

HUMBER 

1  TITLE 

Testability  Analysis  Report 

D00 

DI-T-7199 

I.  OCJCftIP  T  ION/  PURPOSE 

3.1  This  report  documents  the  results  of  the  testability 
requirements,  design  and  evaluation  tasks  of  M1L-STD- 
2165. 

4,  APPROVAU  DATE 

29  January  1985 

8.  OFFICE  OF  PRIMARY 

RESPONSIBILITY 

NAVY-EC 

(.1  DDC  PKQUIRID 

8.  APPROVAL  LIMITATION 

?.  APPLIC  ATION/INT  ERREL  ATIONSMIP 

7.1  These  data  are  to  be  used  to  evaluate  the  level  of 
testability  incorporated  in  a  design. 

7.2  This  DID  may  be  used  for  all  electronic  system  and 
equipment  development  programs. 

7.3  This  DID  satisfies  the  data  requirements  of  Tasks  201, 
202  and  203  of  MIL-STD-2165. 

1 _ 

t.  REFERENCES  (Mondmlory  «•  cited  In 
block  10) 

MIL-STD-2165 

MCSL  NUMBKRIH 

AMSC  NO.  N3425 

10.  PREPARATION  INSTRUCTIONS 


10.1  The  applicable  Issue  of  the  documents  cited  herein,  Including  their  approval 
dates  and  applicable  amendments  and  revisions,  shall  be  as  reflected  In  the 
contract. 

10.2  Contractor's  format  Is  acceptable. 

10.3  The  content  of  the  Testability  Analysis  Report  shall  Include  the  following: 

10.3.1  General 

10.3.1.1  A  brief  description  of  the  system's  functional  operation. 

10.3.1.2  A  brief  description  of  the  functional  operation  of  each  Item. 

10.3.1.3  A  description  of  system  maintenance  and  support  concept. 

10.3.2  Testability  Requirements  Analysis  (MIL-STD-2165,  Task  201) 

10.3.2.1  Description  of  methodology  used  to  trade-off  alternative  diagnostic 
concepts,  including  varying  degrees  of  built-in  test,  automatic  test 
equipment  and  manual  test. 

10.3.2.2  Results  of  diagnostic  trade-offs,  Including  the  Impact  of  each  alternative 
on  readiness,  life  cycle  costs,  manpower  and  training. 
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testability  Analysis  Report 


10.3.2.3  Description  of  the  selected  system  diagnostic  concept  Including  recommended 
testability  requirements  for  the  system  specification. 

10.3.2.4  Description  of  methodology  used  to  allocate  system  testability  requirements 
to  each  item;  recommended  testability  requirements  for  each  item. 

10.3.3 

10.3.3.1  Description  of  system  built-in  test  functional  design  and  system  partitioning 
used  to  enhance  testing. 

10.3.3.2  For  each  item  to  be  included  in  this  analysis,  a  description  of  testability 
features  incorporated  (compatibility,  observability,  controllability, 
partitioning,  etc.),  BIT  functional  design  and  BIT  interfaces  to  system 
BIT  and  to  external  test. 

10.3.3.3  For  each  item  to  be  included  in  the  Inherent  Testability  Assessment, 
recommended  weighting  factors  and  scoring  method  for  each  testability 
criteria  in  the  checklist. 

10.3.3.4  For  each  item  to  be  included  in  the  Inherent  Testability  Assessment,  a 
filled-in  checklist  and  the  calculated  inherent  testability. 

10.3.3.5  Description  of  methodologies,  models  and  tools  to  be  used  in  predicting 
built-in  test  fault  detection  and  fault  isolation  effectiveness. 


Preliminary  Testability  Design  Analysis  (MIL-STD-2165,  Task  202] 


10.3.4 


10.3.4.1  For  each  Item  to  be  included  in  this  analysis,  a  definition  of  predominant 
failure  modes  to  be  tested,  a  prediction  of  built-in  test  fault  detection 
and  fault  isolation  effectiveness  and  identification  of  areas  which  require 
additional  testing. 


10.3.4.2  Prediction  of  built-in  test  fault  detection,  fault  Isolation  and  false  alarm 
characteristics  at  the  system  level. 


10.3.4.3  Estimation  of  costs  associated  with  the  incorporation  of  built-in  test  and 
testability  features,  including  developmental  costs  and  recurring  costs. 
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